[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:27 dist-test-slave-qhrh krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:27 dist-test-slave-qhrh krb5kdc[395](info): set up 2 sockets
Sep 24 11:40:27 dist-test-slave-qhrh krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 11:40:28.765208 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.010s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:40:28 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:40:28.919198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:29.235409 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:29.235641 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:29.235795 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:29.242945 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:29.243024 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:29.243110 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:29.243178 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:29.243244 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:29.251922 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:29.253032 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:29.254179 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:29.265192 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:29.265434 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:29.265327 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:29.266589 408 server_base.cc:1034] running on GCE node
I20260924 11:40:29.267310 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:29.268556 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:29.269809 408 hybrid_clock.cc:648] HybridClock initialized: now 1790250029269719 us; error 69 us; skew 500 ppm
I20260924 11:40:29.270085 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:29 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:29.273105 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:29.274349 408 webserver.cc:460] Webserver started at http://127.0.97.62:40417/ using document root <none> and password file <none>
I20260924 11:40:29.274914 408 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:29.275012 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:29.275224 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:29.278322 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5"
format_stamp: "Formatted at 2026-09-24 11:40:29 on dist-test-slave-qhrh"
server_key: "b4e1b5e36b280d3d37b2c68a18cf9cd1"
server_key_iv: "86a9bea50c1d92fb8d682da65b1d4ca0"
server_key_version: "encryptionkey@0"
I20260924 11:40:29.278940 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5"
format_stamp: "Formatted at 2026-09-24 11:40:29 on dist-test-slave-qhrh"
server_key: "b4e1b5e36b280d3d37b2c68a18cf9cd1"
server_key_iv: "86a9bea50c1d92fb8d682da65b1d4ca0"
server_key_version: "encryptionkey@0"
I20260924 11:40:29.283397 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 11:40:29.286386 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:29.287501 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:29.287611 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5"
format_stamp: "Formatted at 2026-09-24 11:40:29 on dist-test-slave-qhrh"
server_key: "b4e1b5e36b280d3d37b2c68a18cf9cd1"
server_key_iv: "86a9bea50c1d92fb8d682da65b1d4ca0"
server_key_version: "encryptionkey@0"
I20260924 11:40:29.287739 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:29.302022 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:29.305881 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:29.306111 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:29.325012 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42823
I20260924 11:40:29.325052 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42823 every 8 connection(s)
I20260924 11:40:29.326251 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:29.329864 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:29.331784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 408
I20260924 11:40:29.332058 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:29.333855 388 external_mini_cluster.cc:1428] Setting key 9ecb9fc9410227171d98eca032e5b6fb
I20260924 11:40:29.339818 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5: Bootstrap starting.
I20260924 11:40:29.342866 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:29.343755 475 log.cc:824] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:29.346163 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5: No bootstrap required, opened a new log
I20260924 11:40:29.355651 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42823 } }
I20260924 11:40:29.356060 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:29.356184 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb944c7e4ca4e43b9e7f252ee5e33b5, State: Initialized, Role: FOLLOWER
I20260924 11:40:29.356686 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42823 } }
I20260924 11:40:29.356878 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:29.357039 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:29.357179 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:29.358922 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42823 } }
I20260924 11:40:29.359352 475 leader_election.cc:302] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeb944c7e4ca4e43b9e7f252ee5e33b5; no voters:
Sep 24 11:40:29 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:29.360378 475 leader_election.cc:288] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:29.361393 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:29.362592 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [term 1 LEADER]: Becoming Leader. State: Replica: aeb944c7e4ca4e43b9e7f252ee5e33b5, State: Running, Role: LEADER
I20260924 11:40:29.363111 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42823 } }
I20260924 11:40:29.363386 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:29.365725 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb944c7e4ca4e43b9e7f252ee5e33b5. Latest consensus state: current_term: 1 leader_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42823 } } }
I20260924 11:40:29.366149 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:29.366719 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb944c7e4ca4e43b9e7f252ee5e33b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42823 } } }
I20260924 11:40:29.366976 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:29.369975 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:29.374289 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:29.375528 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:29.337379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.0.97.62:42823)
0924 11:40:29.338255 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:29.338271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:29.339485 (+ 1214us) server_negotiation.cc:407] Connection header received
0924 11:40:29.348397 (+ 8912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:29.348419 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:29.348848 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:29.349216 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:29.351490 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:29.353540 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:29.354998 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:29.355889 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:29.356094 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:29.361412 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:29.361424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:29.361445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:29.361498 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:29.366463 (+ 4965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:29.367681 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:29.367693 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:29.367703 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:29.368251 (+ 548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:29.368917 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:29.368921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:29.368924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:29.369766 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:29.373116 (+ 3350us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:29.373491 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 11:40:29.373959 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":645,"thread_start_us":430,"threads_started":1}
I20260924 11:40:29.385803 488 catalog_manager.cc:1347] Generated new cluster ID: 685b101150b34862af6df85ee08715d4
I20260924 11:40:29.385916 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:29.397639 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:29.398661 488 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:29.403746 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeb944c7e4ca4e43b9e7f252ee5e33b5: Generated new TSK 0
I20260924 11:40:29.404377 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:40:29.452666 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42823
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:29.763619 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:29.763890 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:29.764060 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:29.771231 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:29.771369 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:29.771538 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:40:29.779940 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:29.781033 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:29.782213 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:29.794648 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:29.794677 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:29.794698 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:29.795445 501 server_base.cc:1034] running on GCE node
I20260924 11:40:29.795964 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:29.796757 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:29.797925 501 hybrid_clock.cc:648] HybridClock initialized: now 1790250029797883 us; error 40 us; skew 500 ppm
I20260924 11:40:29.798178 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:29 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:29.800968 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:40:29.802255 501 webserver.cc:460] Webserver started at http://127.0.97.1:45349/ using document root <none> and password file <none>
I20260924 11:40:29.802855 501 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:29.802970 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:29.803249 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:29.805858 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "70f1fbdf44b942f7b38328f15f42432d"
format_stamp: "Formatted at 2026-09-24 11:40:29 on dist-test-slave-qhrh"
server_key: "41be84fbb4d5db5a414a2158c4b0374e"
server_key_iv: "9a300267eb90e30279a220384206bbc7"
server_key_version: "encryptionkey@0"
I20260924 11:40:29.806978 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "70f1fbdf44b942f7b38328f15f42432d"
format_stamp: "Formatted at 2026-09-24 11:40:29 on dist-test-slave-qhrh"
server_key: "41be84fbb4d5db5a414a2158c4b0374e"
server_key_iv: "9a300267eb90e30279a220384206bbc7"
server_key_version: "encryptionkey@0"
I20260924 11:40:29.811583 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 11:40:29.814579 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:29.815817 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:29.815989 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "70f1fbdf44b942f7b38328f15f42432d"
format_stamp: "Formatted at 2026-09-24 11:40:29 on dist-test-slave-qhrh"
server_key: "41be84fbb4d5db5a414a2158c4b0374e"
server_key_iv: "9a300267eb90e30279a220384206bbc7"
server_key_version: "encryptionkey@0"
I20260924 11:40:29.816202 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:29.834952 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:29.838421 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:29.838678 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:29.839504 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:29.840773 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:29.840876 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:29.841018 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:29.841086 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:29.874070 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45559
I20260924 11:40:29.874099 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45559 every 8 connection(s)
I20260924 11:40:29.875197 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:40:29.876484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 501
I20260924 11:40:29.876679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:40:29.877238 388 external_mini_cluster.cc:1428] Setting key 6b94aed19efff1706b600b72ee9a1d64
Sep 24 11:40:29 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 11:40:29.897039 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:29.879250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48375 (local address 127.0.97.62:42823)
0924 11:40:29.879674 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:29.879678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:29.881082 (+ 1404us) server_negotiation.cc:407] Connection header received
0924 11:40:29.882825 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:29.882830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:29.882921 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:29.883397 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:29.884870 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:29.885498 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:29.886959 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:29.887367 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:29.887503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:29.891427 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:29.891433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:29.891438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:29.891481 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:29.894044 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:29.894757 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:29.894762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:29.894771 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:29.894917 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:29.895294 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:29.895300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:29.895304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:29.895650 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:29.895793 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:29.896358 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 11:40:29.896581 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":228,"threads_started":1}
I20260924 11:40:29.897483 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:29.879786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42823 (local address 127.0.97.1:48375)
0924 11:40:29.880568 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:29.880602 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:29.882557 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:29.883619 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:29.883626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:29.884232 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:29.884715 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:29.884730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:29.885633 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:29.885640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:29.886815 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:29.886826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:29.887591 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:29.887617 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:29.887834 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:29.888520 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:29.888545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:29.891136 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:29.894240 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:29.894246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:29.894255 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:29.894611 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:29.895022 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:29.895025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:29.895027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:29.895174 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:29.895759 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:29.895763 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:29.896008 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 11:40:29.896239 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 11:40:29.896548 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":334,"threads_started":1}
I20260924 11:40:29.898967 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42823
I20260924 11:40:29.899236 629 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:29.899986 629 heartbeater.cc:511] Master 127.0.97.62:42823 requested a full tablet report, sending...
I20260924 11:40:29.901960 440 ts_manager.cc:194] Registered new tserver with Master: 70f1fbdf44b942f7b38328f15f42432d (127.0.97.1:45559)
I20260924 11:40:29.903504 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:40:29.915647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42823
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:30.210608 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:30.210843 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:30.210922 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:30.217476 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:30.217557 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:30.217743 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:40:30.226845 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:30.227929 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:30.229110 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:30.240718 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:30.240939 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:30.240963 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:30.242045 635 server_base.cc:1034] running on GCE node
I20260924 11:40:30.242527 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:30.243234 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:30.244406 635 hybrid_clock.cc:648] HybridClock initialized: now 1790250030244355 us; error 46 us; skew 500 ppm
I20260924 11:40:30.244625 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:30.247527 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:40:30.248718 635 webserver.cc:460] Webserver started at http://127.0.97.2:34163/ using document root <none> and password file <none>
I20260924 11:40:30.249300 635 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:30.249388 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:30.249606 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:30.252027 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "864ccde89aaa46a08abdd3f71204c024"
format_stamp: "Formatted at 2026-09-24 11:40:30 on dist-test-slave-qhrh"
server_key: "0a6307ca55ba81092be17e849f1f2598"
server_key_iv: "cd091c0b7aaa8633f879f7abde57b5a5"
server_key_version: "encryptionkey@0"
I20260924 11:40:30.252780 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "864ccde89aaa46a08abdd3f71204c024"
format_stamp: "Formatted at 2026-09-24 11:40:30 on dist-test-slave-qhrh"
server_key: "0a6307ca55ba81092be17e849f1f2598"
server_key_iv: "cd091c0b7aaa8633f879f7abde57b5a5"
server_key_version: "encryptionkey@0"
I20260924 11:40:30.256927 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 11:40:30.259687 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:30.260789 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:40:30.260937 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "864ccde89aaa46a08abdd3f71204c024"
format_stamp: "Formatted at 2026-09-24 11:40:30 on dist-test-slave-qhrh"
server_key: "0a6307ca55ba81092be17e849f1f2598"
server_key_iv: "cd091c0b7aaa8633f879f7abde57b5a5"
server_key_version: "encryptionkey@0"
I20260924 11:40:30.261531 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:30.286455 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:30.289899 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:30.290138 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:30.290941 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:30.292160 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:30.292260 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:30.292407 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:30.292476 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:30.325695 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46471
I20260924 11:40:30.325713 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46471 every 8 connection(s)
I20260924 11:40:30.327104 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:40:30.328387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 635
I20260924 11:40:30.328601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:40:30.329103 388 external_mini_cluster.cc:1428] Setting key 20492de07f90ab2301cb54aeb5350fb2
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 11:40:30.349920 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.329642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42285 (local address 127.0.97.62:42823)
0924 11:40:30.329966 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:30.329972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:30.331229 (+ 1257us) server_negotiation.cc:407] Connection header received
0924 11:40:30.334027 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:30.334035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:30.334143 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:30.334243 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:30.335955 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.336772 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.338239 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.338642 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.338770 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.342968 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:30.342972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:30.342975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:30.343003 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:30.345755 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.346783 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.346788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.346798 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.347000 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.347543 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.347548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.347551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.347904 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:30.348052 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:30.349067 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0924 11:40:30.349412 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260924 11:40:30.350919 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.330151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42823 (local address 127.0.97.2:42285)
0924 11:40:30.331097 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:30.331136 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:30.333476 (+ 2340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:30.334522 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:30.334533 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:30.335167 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:30.335801 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:30.335820 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.336926 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.336931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:30.338102 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:30.338113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.338859 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.338865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:30.339080 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.339705 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:30.339730 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:30.342764 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:30.345896 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:30.345903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:30.345917 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:30.346646 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:30.347166 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:30.347170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:30.347174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:30.347401 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:30.348147 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:30.348156 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:30.348581 (+ 425us) client_negotiation.cc:769] Sending connection context
0924 11:40:30.348904 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 11:40:30.349387 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":504,"threads_started":1}
I20260924 11:40:30.352548 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42823
I20260924 11:40:30.352835 763 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:30.353430 763 heartbeater.cc:511] Master 127.0.97.62:42823 requested a full tablet report, sending...
I20260924 11:40:30.354691 440 ts_manager.cc:194] Registered new tserver with Master: 864ccde89aaa46a08abdd3f71204c024 (127.0.97.2:46471)
I20260924 11:40:30.355643 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:40:30.370891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42823
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:30.688442 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:30.688644 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:30.688715 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:30.694983 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:30.695060 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:30.695156 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:40:30.703790 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:30.704833 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:30.705981 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:30.717689 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:30.717782 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:30.717754 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:30.718309 768 server_base.cc:1034] running on GCE node
I20260924 11:40:30.718909 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:30.719767 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:30.720973 768 hybrid_clock.cc:648] HybridClock initialized: now 1790250030720942 us; error 59 us; skew 500 ppm
I20260924 11:40:30.721248 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:30.724327 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:40:30.725541 768 webserver.cc:460] Webserver started at http://127.0.97.3:33871/ using document root <none> and password file <none>
I20260924 11:40:30.726146 768 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:30.726279 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:30.726558 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:30.729461 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "9c2b90f87b0e4fa481603c496dea642b"
format_stamp: "Formatted at 2026-09-24 11:40:30 on dist-test-slave-qhrh"
server_key: "263836382f798213a600f8cc41054519"
server_key_iv: "c54048d100543949fafec07832ceb272"
server_key_version: "encryptionkey@0"
I20260924 11:40:30.730176 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9c2b90f87b0e4fa481603c496dea642b"
format_stamp: "Formatted at 2026-09-24 11:40:30 on dist-test-slave-qhrh"
server_key: "263836382f798213a600f8cc41054519"
server_key_iv: "c54048d100543949fafec07832ceb272"
server_key_version: "encryptionkey@0"
I20260924 11:40:30.734611 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:40:30.737571 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:30.738686 768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:40:30.738837 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "9c2b90f87b0e4fa481603c496dea642b"
format_stamp: "Formatted at 2026-09-24 11:40:30 on dist-test-slave-qhrh"
server_key: "263836382f798213a600f8cc41054519"
server_key_iv: "c54048d100543949fafec07832ceb272"
server_key_version: "encryptionkey@0"
I20260924 11:40:30.739001 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:30.762333 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:30.766321 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:30.766592 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:30.767386 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:30.768657 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:30.768759 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:30.768887 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:30.768955 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:30.800840 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32837
I20260924 11:40:30.800906 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32837 every 8 connection(s)
I20260924 11:40:30.801914 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:40:30.804464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 768
I20260924 11:40:30.804642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:40:30.805190 388 external_mini_cluster.cc:1428] Setting key 0c121c120553a8398c2ad2e66b2f6f33
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:30.819543 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.804075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36767 (local address 127.0.97.62:42823)
0924 11:40:30.804260 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:30.804264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:30.805505 (+ 1241us) server_negotiation.cc:407] Connection header received
0924 11:40:30.807724 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:30.807730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:30.807815 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:30.807920 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:30.809210 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.809915 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.811275 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.811670 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.811797 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.814775 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:30.814781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:30.814784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:30.814809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:30.816946 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.817586 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.817597 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.817600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.817701 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.818042 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.818046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.818049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.818262 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:30.818426 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:30.818992 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 11:40:30.819240 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:40:30.820513 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.804654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42823 (local address 127.0.97.3:36767)
0924 11:40:30.805367 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:30.805400 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:30.807473 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:30.808135 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:30.808179 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:30.808599 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:30.809038 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:30.809070 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.810061 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.810066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:30.811137 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:30.811146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.811869 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.811879 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:30.812113 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.812708 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:30.812727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:30.814579 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:30.817089 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:30.817096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:30.817108 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:30.817475 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:30.817791 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:30.817796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:30.817799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:30.817940 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:30.818401 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:30.818407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:30.818663 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 11:40:30.818880 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 11:40:30.819354 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":265,"threads_started":1}
I20260924 11:40:30.821667 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42823
I20260924 11:40:30.821918 896 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:30.822500 896 heartbeater.cc:511] Master 127.0.97.62:42823 requested a full tablet report, sending...
I20260924 11:40:30.823644 440 ts_manager.cc:194] Registered new tserver with Master: 9c2b90f87b0e4fa481603c496dea642b (127.0.97.3:32837)
I20260924 11:40:30.824429 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36767
I20260924 11:40:30.832325 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:30.874302 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.860343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.97.1:45559)
0924 11:40:30.860887 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:30.860894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:30.860978 (+ 84us) server_negotiation.cc:407] Connection header received
0924 11:40:30.861102 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:30.861109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:30.861293 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:30.861440 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:30.862720 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.863724 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.865059 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.865625 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.865725 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.869273 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:30.869282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:30.869292 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:30.869322 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:30.871735 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.872385 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.872392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.872394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.872504 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.872848 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.872860 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.872862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.873136 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:30.873457 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:30.873697 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 11:40:30.873970 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":285,"threads_started":1}
W20260924 11:40:30.875867 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50216
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260924 11:40:30.906024 629 heartbeater.cc:503] Master 127.0.97.62:42823 was elected leader, sending a full tablet report...
Sep 24 11:40:30 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:30.912653 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.900570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.97.1:45559)
0924 11:40:30.900753 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:30.900758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:30.900878 (+ 120us) server_negotiation.cc:407] Connection header received
0924 11:40:30.900986 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:30.900989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:30.901058 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:30.901130 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:30.902275 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.902916 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.903988 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.904430 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.904516 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.907999 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:30.908015 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:30.908018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:30.908042 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:30.910121 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.910650 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.910653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.910655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.910752 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.911031 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.911034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.911036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.911240 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:30.911400 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:30.911669 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 11:40:30.912314 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:40:30.913936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 501
I20260924 11:40:30.925441 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:30.914752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.97.1:45559)
0924 11:40:30.914924 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:30.914931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:30.915209 (+ 278us) server_negotiation.cc:407] Connection header received
0924 11:40:30.915384 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:30.915387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:30.915434 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:30.915524 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:30.916695 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.917325 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.918509 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:30.918888 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:30.919006 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:30.921018 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:30.921028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:30.921031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:30.921056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:30.923144 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.923727 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.923734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.923738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.923850 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:30.924245 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:30.924250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:30.924253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:30.924473 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:30.924679 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:30.924958 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 11:40:30.925135 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:40:30.925994 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:31.003660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 501
I20260924 11:40:31.023239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 635
Sep 24 11:40:31 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:40:31.037989 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:31.024191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.97.2:46471)
0924 11:40:31.024747 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:31.024753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:31.024778 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:40:31.024834 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:31.024841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:31.025038 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:31.025164 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:31.026296 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:31.027314 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:31.028586 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:31.029096 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:31.029185 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:31.032970 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:31.032979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:31.032991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:31.033026 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:31.035444 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:31.036012 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:31.036019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:31.036021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:31.036127 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:31.036625 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:31.036628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:31.036630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:31.036993 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:31.037205 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:31.037445 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 11:40:31.037620 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
I20260924 11:40:31.039187 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:31.116204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 635
I20260924 11:40:31.138173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 768
Sep 24 11:40:31 dist-test-slave-qhrh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:40:31.152602 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:31.139140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.0.97.3:32837)
0924 11:40:31.139782 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:31.139789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:31.139816 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:40:31.139871 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:31.139877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:31.140074 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:31.140248 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:31.141314 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:31.142435 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:31.143617 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:31.144114 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:31.144360 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:31.148210 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:31.148228 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:31.148240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:31.148266 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:31.150496 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:31.150996 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:31.151001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:31.151003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:31.151099 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:31.151351 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:31.151353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:31.151355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:31.151660 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:31.151876 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:31.152084 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 11:40:31.152315 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":370,"threads_started":1}
I20260924 11:40:31.153708 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:31.231395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 768
I20260924 11:40:31.248916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 408
I20260924 11:40:31.250070 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:31.353031 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 408
2026-09-24T11:40:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3655 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:33 dist-test-slave-qhrh krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:33 dist-test-slave-qhrh krb5kdc[917](info): set up 2 sockets
Sep 24 11:40:33 dist-test-slave-qhrh krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 11:40:34.327364 388 mini_kdc.cc:121] Time spent starting KDC: real 2.934s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 11:40:34.506137 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.179s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 11:40:34.719911 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.214s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/test-user.keytab.
W20260924 11:40:35.283365 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.487s user 0.000s sys 0.005s
Sep 24 11:40:35 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260924 11:40:35.973557 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.647s user 0.000s sys 0.007s
I20260924 11:40:35.974654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45273
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:36.301817 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:36.302155 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:36.302316 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:36.309508 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:36.309638 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:36.309736 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:36.309819 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:36.309900 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:36.319450 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:36.320815 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:36.322172 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:36.333763 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:36.333789 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:36.333753 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:36.334687 930 server_base.cc:1034] running on GCE node
I20260924 11:40:36.335350 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:36.336621 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:36.337810 930 hybrid_clock.cc:648] HybridClock initialized: now 1790250036337771 us; error 30 us; skew 500 ppm
I20260924 11:40:36.338143 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:36 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:36.348971 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:36.350306 930 webserver.cc:460] Webserver started at http://127.0.97.62:32921/ using document root <none> and password file <none>
I20260924 11:40:36.350919 930 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:36.351042 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:36.351341 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:36.354116 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53"
format_stamp: "Formatted at 2026-09-24 11:40:36 on dist-test-slave-qhrh"
server_key: "9fa4bba8d4ac32ca1d421f8311282682"
server_key_iv: "353dfa4b0e882e09a9cab24b95f1e67e"
server_key_version: "encryptionkey@0"
I20260924 11:40:36.354858 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53"
format_stamp: "Formatted at 2026-09-24 11:40:36 on dist-test-slave-qhrh"
server_key: "9fa4bba8d4ac32ca1d421f8311282682"
server_key_iv: "353dfa4b0e882e09a9cab24b95f1e67e"
server_key_version: "encryptionkey@0"
I20260924 11:40:36.359580 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:40:36.362754 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:36.363940 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:40:36.364111 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53"
format_stamp: "Formatted at 2026-09-24 11:40:36 on dist-test-slave-qhrh"
server_key: "9fa4bba8d4ac32ca1d421f8311282682"
server_key_iv: "353dfa4b0e882e09a9cab24b95f1e67e"
server_key_version: "encryptionkey@0"
I20260924 11:40:36.364308 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:36.384827 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:36.388464 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:36.388710 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:36.405869 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45273
I20260924 11:40:36.405902 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45273 every 8 connection(s)
I20260924 11:40:36.407107 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:36.409540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 930
I20260924 11:40:36.409767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:36.410553 388 external_mini_cluster.cc:1428] Setting key b58e9182fe8618e0376835a93b020ca8
I20260924 11:40:36.411051 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:36.424849 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53: Bootstrap starting.
Sep 24 11:40:36 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:36.427976 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:36.429080 997 log.cc:824] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:36.431592 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53: No bootstrap required, opened a new log
I20260924 11:40:36.433869 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:36.412480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.0.97.62:45273)
0924 11:40:36.413207 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:36.413217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:36.413253 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:40:36.415376 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:36.415396 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:36.415798 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:36.416115 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:36.417855 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:36.419073 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:36.421642 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:36.422559 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:36.422693 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:36.427065 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:36.427072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:36.427084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:36.427114 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:36.430152 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:36.430756 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:36.430762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:36.430767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:36.430908 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:36.431287 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:36.431296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:36.431298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:36.431785 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:36.432076 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:36.432582 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 11:40:36.432945 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":342,"threads_started":1}
I20260924 11:40:36.439823 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45273 } }
I20260924 11:40:36.440186 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:36.440299 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2d1eb1bb6e4a18b81c0a20dd6a2c53, State: Initialized, Role: FOLLOWER
I20260924 11:40:36.440866 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45273 } }
I20260924 11:40:36.441123 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:36.441242 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:36.441396 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:36.450038 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45273 } }
I20260924 11:40:36.450546 997 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2d1eb1bb6e4a18b81c0a20dd6a2c53; no voters:
I20260924 11:40:36.451128 997 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:36.451224 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:36.451792 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2d1eb1bb6e4a18b81c0a20dd6a2c53, State: Running, Role: LEADER
I20260924 11:40:36.452402 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45273 } }
I20260924 11:40:36.454028 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:36.455482 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45273 } } }
I20260924 11:40:36.455806 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:36.457026 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2d1eb1bb6e4a18b81c0a20dd6a2c53. Latest consensus state: current_term: 1 leader_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d1eb1bb6e4a18b81c0a20dd6a2c53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45273 } } }
I20260924 11:40:36.457306 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:36.457887 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:36.463653 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:36.471769 1007 catalog_manager.cc:1347] Generated new cluster ID: 09d85ed38a644ea0b4bd1a5d5a980d2c
I20260924 11:40:36.471873 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:36.488250 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:36.489506 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:36.496258 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2d1eb1bb6e4a18b81c0a20dd6a2c53: Generated new TSK 0
I20260924 11:40:36.496986 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260924 11:40:36.806339 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.296s user 0.001s sys 0.009s
I20260924 11:40:36.807168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45273
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:37.102138 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:37.102375 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:37.102540 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:37.109395 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:37.109508 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:37.109684 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:40:37.118492 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:37.119546 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:37.120748 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:37.132917 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:37.133059 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:37.133057 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:37.133668 1023 server_base.cc:1034] running on GCE node
I20260924 11:40:37.134202 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:37.135012 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:37.136234 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790250037136195 us; error 44 us; skew 500 ppm
I20260924 11:40:37.136518 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:37 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:37.139374 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:40:37.140609 1023 webserver.cc:460] Webserver started at http://127.0.97.1:40451/ using document root <none> and password file <none>
I20260924 11:40:37.141170 1023 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:37.141258 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:37.141484 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:37.144033 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "5fd6fd7800a54dcb94f6d431265e860d"
format_stamp: "Formatted at 2026-09-24 11:40:37 on dist-test-slave-qhrh"
server_key: "68f22ff00617e8755041b77d4c266524"
server_key_iv: "aeced94faa4870532ba4ee0662d6f40b"
server_key_version: "encryptionkey@0"
I20260924 11:40:37.144757 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5fd6fd7800a54dcb94f6d431265e860d"
format_stamp: "Formatted at 2026-09-24 11:40:37 on dist-test-slave-qhrh"
server_key: "68f22ff00617e8755041b77d4c266524"
server_key_iv: "aeced94faa4870532ba4ee0662d6f40b"
server_key_version: "encryptionkey@0"
I20260924 11:40:37.149223 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:40:37.152071 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:37.153290 1023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 11:40:37.153404 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "5fd6fd7800a54dcb94f6d431265e860d"
format_stamp: "Formatted at 2026-09-24 11:40:37 on dist-test-slave-qhrh"
server_key: "68f22ff00617e8755041b77d4c266524"
server_key_iv: "aeced94faa4870532ba4ee0662d6f40b"
server_key_version: "encryptionkey@0"
I20260924 11:40:37.153546 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:37.166476 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:37.170301 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:37.170495 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:37.171244 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:37.172410 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:37.172479 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:37.172557 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:37.172603 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:37.203194 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42747
I20260924 11:40:37.203282 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42747 every 8 connection(s)
I20260924 11:40:37.204394 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:40:37.210817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1023
I20260924 11:40:37.211028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:40:37.211769 388 external_mini_cluster.cc:1428] Setting key 42d805da2c3dc25f7a6b9d57660c4f0e
Sep 24 11:40:37 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:37.225400 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:37.206660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59077 (local address 127.0.97.62:45273)
0924 11:40:37.207029 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:37.207033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:37.207962 (+ 929us) server_negotiation.cc:407] Connection header received
0924 11:40:37.209741 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:37.209746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:37.209833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:37.209944 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:37.211683 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.212607 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.214462 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.214825 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.214914 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:37.219018 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:37.219022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:37.219025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:37.219053 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:37.222457 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:37.223082 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:37.223090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:37.223097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:37.223237 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:37.223604 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:37.223611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:37.223617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:37.223968 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:37.224112 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:37.224749 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 11:40:37.225000 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":181,"threads_started":1}
I20260924 11:40:37.226004 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:37.207166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45273 (local address 127.0.97.1:59077)
0924 11:40:37.207816 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:37.207854 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:37.209497 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:37.210135 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:37.210142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:37.210757 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:37.211512 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:37.211536 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.212750 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.212758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:37.214329 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:37.214343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.215037 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.215046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:37.215366 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:37.216075 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:37.216109 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:37.218825 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:37.222578 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:37.222584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:37.222594 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:37.222969 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:37.223350 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:37.223353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:37.223355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:37.223502 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:37.224077 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:37.224082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:37.224411 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 11:40:37.224610 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 11:40:37.225044 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260924 11:40:37.227365 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45273
I20260924 11:40:37.227662 1151 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:37.228520 1151 heartbeater.cc:511] Master 127.0.97.62:45273 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:40:37.230576 962 ts_manager.cc:194] Registered new tserver with Master: 5fd6fd7800a54dcb94f6d431265e860d (127.0.97.1:42747)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 11:40:37.231980 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:40:37.251080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45273
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:37.547966 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:37.548265 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:37.548449 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:37.555409 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:37.555544 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:37.555708 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:40:37.564958 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:37.566093 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:37.567361 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:37.579602 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:37.579680 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:37.579668 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:37.585429 1157 server_base.cc:1034] running on GCE node
I20260924 11:40:37.586217 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:37.587092 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:37.588321 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790250037588266 us; error 49 us; skew 500 ppm
I20260924 11:40:37.588624 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:37 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:37.591542 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:40:37.592991 1157 webserver.cc:460] Webserver started at http://127.0.97.2:34677/ using document root <none> and password file <none>
I20260924 11:40:37.593629 1157 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:37.593744 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:37.594031 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:37.596745 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "68ff49622e73492aa5da588a1707e7b4"
format_stamp: "Formatted at 2026-09-24 11:40:37 on dist-test-slave-qhrh"
server_key: "4c21c32a9b0169906b64e9c7e441207b"
server_key_iv: "d8e5bd8d244409f93f12f5148b5f36c8"
server_key_version: "encryptionkey@0"
I20260924 11:40:37.597458 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "68ff49622e73492aa5da588a1707e7b4"
format_stamp: "Formatted at 2026-09-24 11:40:37 on dist-test-slave-qhrh"
server_key: "4c21c32a9b0169906b64e9c7e441207b"
server_key_iv: "d8e5bd8d244409f93f12f5148b5f36c8"
server_key_version: "encryptionkey@0"
I20260924 11:40:37.601840 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 11:40:37.605123 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:37.606230 1157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:40:37.606385 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "68ff49622e73492aa5da588a1707e7b4"
format_stamp: "Formatted at 2026-09-24 11:40:37 on dist-test-slave-qhrh"
server_key: "4c21c32a9b0169906b64e9c7e441207b"
server_key_iv: "d8e5bd8d244409f93f12f5148b5f36c8"
server_key_version: "encryptionkey@0"
I20260924 11:40:37.606556 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:37.625265 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:37.629633 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:37.629878 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:37.630692 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:37.631803 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:37.631867 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:37.632032 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:37.632083 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:37.665241 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40423
I20260924 11:40:37.665288 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40423 every 8 connection(s)
I20260924 11:40:37.666414 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:40:37.674042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1157
I20260924 11:40:37.674191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:40:37.674606 388 external_mini_cluster.cc:1428] Setting key 660be900b12b43ba414ec3edce6b0a51
Sep 24 11:40:37 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:37.684670 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:37.668808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46169 (local address 127.0.97.62:45273)
0924 11:40:37.668961 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:37.668965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:37.669966 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 11:40:37.671617 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:37.671623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:37.671710 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:37.671810 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:37.673219 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.673907 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.675640 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.676212 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.676343 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:37.679519 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:37.679523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:37.679527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:37.679555 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:37.681949 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:37.682625 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:37.682632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:37.682635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:37.682752 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:37.683257 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:37.683262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:37.683265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:37.683498 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:37.683628 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:37.684170 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 11:40:37.684361 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:40:37.685395 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:37.669227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45273 (local address 127.0.97.2:46169)
0924 11:40:37.669822 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:37.669859 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:37.671374 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:37.672016 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:37.672024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:37.672574 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:37.673063 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:37.673079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.674058 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.674070 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:37.675499 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:37.675513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:37.676418 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:37.676425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:37.676633 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:37.677258 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:37.677280 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:37.679316 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:37.682088 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:37.682096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:37.682125 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:37.682513 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:37.682856 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:37.682861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:37.682865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:37.683079 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:37.683615 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:37.683621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:37.683866 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 11:40:37.684084 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 11:40:37.684519 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":242,"threads_started":1}
I20260924 11:40:37.686808 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45273
I20260924 11:40:37.687074 1285 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:37.687767 1285 heartbeater.cc:511] Master 127.0.97.62:45273 requested a full tablet report, sending...
I20260924 11:40:37.689069 962 ts_manager.cc:194] Registered new tserver with Master: 68ff49622e73492aa5da588a1707e7b4 (127.0.97.2:40423)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:40:37.689859 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:40:37.709612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45273
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:38.002609 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:38.002846 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:38.002935 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:38.009833 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:38.009943 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:38.010076 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:40:38.019091 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:38.020117 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:38.021319 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:38.033509 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:38.033542 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:38.033715 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:38.034147 1290 server_base.cc:1034] running on GCE node
I20260924 11:40:38.034617 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:38.035337 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:38.036516 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790250038036489 us; error 28 us; skew 500 ppm
I20260924 11:40:38.036751 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:38.039450 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:40:38.040820 1290 webserver.cc:460] Webserver started at http://127.0.97.3:41853/ using document root <none> and password file <none>
I20260924 11:40:38.041371 1290 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:38.041461 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:38.041675 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:38.044348 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "e5d94a7624994956bc47372d22d3d920"
format_stamp: "Formatted at 2026-09-24 11:40:38 on dist-test-slave-qhrh"
server_key: "21ee6a69a36bddbf005a245828df9e9c"
server_key_iv: "f9e129d6e1cb9b7ed0ac21a6f3e90825"
server_key_version: "encryptionkey@0"
I20260924 11:40:38.045017 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e5d94a7624994956bc47372d22d3d920"
format_stamp: "Formatted at 2026-09-24 11:40:38 on dist-test-slave-qhrh"
server_key: "21ee6a69a36bddbf005a245828df9e9c"
server_key_iv: "f9e129d6e1cb9b7ed0ac21a6f3e90825"
server_key_version: "encryptionkey@0"
I20260924 11:40:38.049505 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:40:38.052558 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:38.053709 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:40:38.053828 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "e5d94a7624994956bc47372d22d3d920"
format_stamp: "Formatted at 2026-09-24 11:40:38 on dist-test-slave-qhrh"
server_key: "21ee6a69a36bddbf005a245828df9e9c"
server_key_iv: "f9e129d6e1cb9b7ed0ac21a6f3e90825"
server_key_version: "encryptionkey@0"
I20260924 11:40:38.053993 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:38.066542 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:38.070376 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:38.070626 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:38.071451 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:38.072629 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:38.072700 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:38.072853 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:38.072907 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:38.104102 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36025
I20260924 11:40:38.104178 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36025 every 8 connection(s)
I20260924 11:40:38.105207 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:40:38.112612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1290
I20260924 11:40:38.112766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:40:38.113212 388 external_mini_cluster.cc:1428] Setting key 0bc440438941f7952a700e7202f5b4b6
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:38.123117 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.107490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54537 (local address 127.0.97.62:45273)
0924 11:40:38.107649 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.107653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.108694 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 11:40:38.110327 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.110331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.110394 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.110489 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.111711 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.112764 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.114485 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.114958 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.115086 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.118492 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.118496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.118499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.118529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.120823 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.121420 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.121423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.121425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.121546 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.121881 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.121883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.121885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.122094 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.122220 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.122752 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.122893 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:40:38.123934 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.107892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45273 (local address 127.0.97.3:54537)
0924 11:40:38.108556 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:38.108589 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:38.110119 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:38.110671 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:38.110677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:38.111104 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:38.111578 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.111594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.112931 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.112935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.114294 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.114306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.115154 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.115162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.115374 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.115984 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:38.116004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:38.118325 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:38.120933 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:38.120939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:38.120948 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:38.121301 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:38.121643 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:38.121646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:38.121648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:38.121798 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:38.122214 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:38.122219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:38.122471 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 11:40:38.122668 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 11:40:38.122965 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":310,"threads_started":1}
I20260924 11:40:38.125092 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45273
I20260924 11:40:38.125305 1418 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:38.125849 1418 heartbeater.cc:511] Master 127.0.97.62:45273 requested a full tablet report, sending...
I20260924 11:40:38.127285 962 ts_manager.cc:194] Registered new tserver with Master: e5d94a7624994956bc47372d22d3d920 (127.0.97.3:36025)
I20260924 11:40:38.127916 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:40:38.128306 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54537
I20260924 11:40:38.145718 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.134137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.97.62:45273)
0924 11:40:38.134279 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.134283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.134854 (+ 571us) server_negotiation.cc:407] Connection header received
0924 11:40:38.135040 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.135045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.135107 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.135199 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.136551 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.137207 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.138789 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.139146 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.139270 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.141173 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.141179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.141183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.141216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.143231 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.143806 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.143813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.143816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.143931 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.144304 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.144310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.144313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.144556 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.144766 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.145196 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.145403 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:40:38.151254 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:40:38.157904 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:38.181177 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.169167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40423 (local address 127.0.0.1:60724)
0924 11:40:38.173282 (+ 4115us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:38.173310 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:38.173412 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:38.174030 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:38.174037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:38.174066 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:38.174395 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.174407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.176839 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.176846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.178319 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.178333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.180042 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.180048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.180475 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.180531 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 11:40:38.180649 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 11:40:38.180799 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3602,"mutex_wait_us":1,"thread_start_us":3463,"threads_started":1}
I20260924 11:40:38.181857 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.169600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.97.2:40423)
0924 11:40:38.172612 (+ 3012us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.172622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.173537 (+ 915us) server_negotiation.cc:407] Connection header received
0924 11:40:38.173630 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.173638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.173831 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.174009 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:38.174576 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.176699 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.178471 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.179862 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.179984 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.180212 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.181369 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.181536 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2843,"thread_start_us":2724,"threads_started":1}
I20260924 11:40:38.182281 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.169278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.0.97.3:36025)
0924 11:40:38.174948 (+ 5670us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.174956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.174984 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:40:38.175058 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.175063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.175245 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.176157 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:38.176590 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.178152 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.179813 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.181110 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.181233 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.181666 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.181735 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.181872 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5215,"thread_start_us":5071,"threads_started":1}
I20260924 11:40:38.181972 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.168762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36025 (local address 127.0.0.1:34040)
0924 11:40:38.169467 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:38.169516 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:38.169661 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:38.175427 (+ 5766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:38.175431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:38.175466 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:38.176363 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.176374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.178306 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.178311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.179660 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.179670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.181287 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.181293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.181462 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.181493 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:40:38.181566 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 11:40:38.181692 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":582,"threads_started":1}
I20260924 11:40:38.183336 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.173290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.0.97.1:42747)
0924 11:40:38.174883 (+ 1593us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.174895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.175383 (+ 488us) server_negotiation.cc:407] Connection header received
0924 11:40:38.175578 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.175585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.175724 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.175895 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:38.176754 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.178201 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.180709 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.182072 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.182214 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.182407 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.182920 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.183089 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":1323,"threads_started":1}
I20260924 11:40:38.185657 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.173878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42747 (local address 127.0.0.1:60306)
0924 11:40:38.175272 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:38.175285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:38.175421 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:38.175878 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:38.175882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:38.175905 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:38.176572 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.176582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.178373 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.178376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.180549 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.180561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.182555 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.182563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.182726 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.182758 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:40:38.182854 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:40:38.182976 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"mutex_wait_us":35,"thread_start_us":1182,"threads_started":1}
I20260924 11:40:38.185364 1220 tablet_service.cc:1467] Processing CreateTablet for tablet c393aa20adb34e088b9c3a343d4ec15d (DEFAULT_TABLE table=test-table [id=ff20211657b74b56bc0eaa79a199a8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:38.185777 1353 tablet_service.cc:1467] Processing CreateTablet for tablet c393aa20adb34e088b9c3a343d4ec15d (DEFAULT_TABLE table=test-table [id=ff20211657b74b56bc0eaa79a199a8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:38.186767 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c393aa20adb34e088b9c3a343d4ec15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:38.187110 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c393aa20adb34e088b9c3a343d4ec15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:38.189384 1086 tablet_service.cc:1467] Processing CreateTablet for tablet c393aa20adb34e088b9c3a343d4ec15d (DEFAULT_TABLE table=test-table [id=ff20211657b74b56bc0eaa79a199a8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:38.190546 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c393aa20adb34e088b9c3a343d4ec15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:38.196760 1436 tablet_bootstrap.cc:492] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: Bootstrap starting.
I20260924 11:40:38.199733 1436 tablet_bootstrap.cc:654] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:38.200454 1437 tablet_bootstrap.cc:492] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: Bootstrap starting.
I20260924 11:40:38.201493 1436 log.cc:824] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:38.203845 1437 tablet_bootstrap.cc:654] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:38.204883 1437 log.cc:824] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:38.206277 1436 tablet_bootstrap.cc:492] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: No bootstrap required, opened a new log
I20260924 11:40:38.206633 1436 ts_tablet_manager.cc:1397] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 11:40:38.217439 1439 tablet_bootstrap.cc:492] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: Bootstrap starting.
I20260924 11:40:38.219704 1437 tablet_bootstrap.cc:492] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: No bootstrap required, opened a new log
I20260924 11:40:38.219620 1436 raft_consensus.cc:348] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.219978 1437 ts_tablet_manager.cc:1397] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.006s
I20260924 11:40:38.220038 1436 raft_consensus.cc:374] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:38.220197 1436 raft_consensus.cc:729] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d94a7624994956bc47372d22d3d920, State: Initialized, Role: FOLLOWER
I20260924 11:40:38.220405 1439 tablet_bootstrap.cc:654] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:38.220821 1436 consensus_queue.cc:260] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.221496 1439 log.cc:824] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:38.221927 1418 heartbeater.cc:503] Master 127.0.97.62:45273 was elected leader, sending a full tablet report...
I20260924 11:40:38.222883 1436 ts_tablet_manager.cc:1428] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260924 11:40:38.226491 1439 tablet_bootstrap.cc:492] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: No bootstrap required, opened a new log
I20260924 11:40:38.226730 1439 ts_tablet_manager.cc:1397] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:40:38.229169 1437 raft_consensus.cc:348] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.229580 1437 raft_consensus.cc:374] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:38.229730 1437 raft_consensus.cc:729] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ff49622e73492aa5da588a1707e7b4, State: Initialized, Role: FOLLOWER
I20260924 11:40:38.230299 1437 consensus_queue.cc:260] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.231581 1285 heartbeater.cc:503] Master 127.0.97.62:45273 was elected leader, sending a full tablet report...
I20260924 11:40:38.231918 1437 ts_tablet_manager.cc:1428] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 11:40:38.234545 1151 heartbeater.cc:503] Master 127.0.97.62:45273 was elected leader, sending a full tablet report...
I20260924 11:40:38.236212 1439 raft_consensus.cc:348] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.236568 1439 raft_consensus.cc:374] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:38.236657 1439 raft_consensus.cc:729] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd6fd7800a54dcb94f6d431265e860d, State: Initialized, Role: FOLLOWER
I20260924 11:40:38.237114 1439 consensus_queue.cc:260] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.238245 1439 ts_tablet_manager.cc:1428] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260924 11:40:38.356633 1419 tablet.cc:2369] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:40:38.418287 1286 tablet.cc:2369] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:40:38.456388 1152 tablet.cc:2369] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:40:38.664924 1443 raft_consensus.cc:482] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:38.665326 1443 raft_consensus.cc:504] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.667372 1443 leader_election.cc:288] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5d94a7624994956bc47372d22d3d920 (127.0.97.3:36025), 5fd6fd7800a54dcb94f6d431265e860d (127.0.97.1:42747)
I20260924 11:40:38.675217 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.667808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60705 (local address 127.0.97.1:42747)
0924 11:40:38.668043 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.668050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.669539 (+ 1489us) server_negotiation.cc:407] Connection header received
0924 11:40:38.669744 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.669748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.669796 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.669879 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:38.670379 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.670998 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.672938 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.674292 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.674414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.674523 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.674847 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.674975 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:40:38.675293 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.667781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42747 (local address 127.0.97.2:60705)
0924 11:40:38.669440 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:38.669459 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:38.669595 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:38.669893 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:38.669897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:38.669916 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:38.670223 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.670230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.671108 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.671118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.672784 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.672795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.674514 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.674520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.674688 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.674720 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:40:38.674805 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 11:40:38.674918 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"mutex_wait_us":89,"thread_start_us":1385,"threads_started":1}
I20260924 11:40:38.675876 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.668107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51971 (local address 127.0.97.3:36025)
0924 11:40:38.668301 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.668309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.668798 (+ 489us) server_negotiation.cc:407] Connection header received
0924 11:40:38.669026 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.669032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.669107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.669198 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:38.669937 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.670906 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.673861 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.674842 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.674960 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.675044 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.675546 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.675685 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:40:38.676435 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.667669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36025 (local address 127.0.97.2:51971)
0924 11:40:38.668677 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:38.668699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:38.668867 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:38.669356 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:38.669360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:38.669383 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:38.669783 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.669794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.671070 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.671074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.673266 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:38.673278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.675185 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.675190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:38.675378 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.675405 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 11:40:38.676044 (+ 639us) client_negotiation.cc:240] Negotiation successful
0924 11:40:38.676180 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":64,"thread_start_us":697,"threads_started":1}
I20260924 11:40:38.676515 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c393aa20adb34e088b9c3a343d4ec15d" candidate_uuid: "68ff49622e73492aa5da588a1707e7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd6fd7800a54dcb94f6d431265e860d" is_pre_election: true
I20260924 11:40:38.676993 1106 raft_consensus.cc:2454] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ff49622e73492aa5da588a1707e7b4 in term 0.
I20260924 11:40:38.677563 1173 leader_election.cc:302] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd6fd7800a54dcb94f6d431265e860d, 68ff49622e73492aa5da588a1707e7b4; no voters:
I20260924 11:40:38.677927 1443 raft_consensus.cc:2781] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:38.678056 1443 raft_consensus.cc:482] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:38.678138 1443 raft_consensus.cc:3037] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:38.678035 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c393aa20adb34e088b9c3a343d4ec15d" candidate_uuid: "68ff49622e73492aa5da588a1707e7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d94a7624994956bc47372d22d3d920" is_pre_election: true
I20260924 11:40:38.678462 1373 raft_consensus.cc:2454] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ff49622e73492aa5da588a1707e7b4 in term 0.
I20260924 11:40:38.681006 1443 raft_consensus.cc:504] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.681420 1443 leader_election.cc:288] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [CANDIDATE]: Term 1 election: Requested vote from peers e5d94a7624994956bc47372d22d3d920 (127.0.97.3:36025), 5fd6fd7800a54dcb94f6d431265e860d (127.0.97.1:42747)
I20260924 11:40:38.681792 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c393aa20adb34e088b9c3a343d4ec15d" candidate_uuid: "68ff49622e73492aa5da588a1707e7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd6fd7800a54dcb94f6d431265e860d"
I20260924 11:40:38.681854 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c393aa20adb34e088b9c3a343d4ec15d" candidate_uuid: "68ff49622e73492aa5da588a1707e7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d94a7624994956bc47372d22d3d920"
I20260924 11:40:38.681988 1106 raft_consensus.cc:3037] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:38.682025 1373 raft_consensus.cc:3037] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:38.684286 1106 raft_consensus.cc:2454] T c393aa20adb34e088b9c3a343d4ec15d P 5fd6fd7800a54dcb94f6d431265e860d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ff49622e73492aa5da588a1707e7b4 in term 1.
I20260924 11:40:38.684670 1173 leader_election.cc:302] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd6fd7800a54dcb94f6d431265e860d, 68ff49622e73492aa5da588a1707e7b4; no voters:
I20260924 11:40:38.684713 1373 raft_consensus.cc:2454] T c393aa20adb34e088b9c3a343d4ec15d P e5d94a7624994956bc47372d22d3d920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ff49622e73492aa5da588a1707e7b4 in term 1.
I20260924 11:40:38.685024 1443 raft_consensus.cc:2781] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:38.685371 1443 raft_consensus.cc:686] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 68ff49622e73492aa5da588a1707e7b4, State: Running, Role: LEADER
I20260924 11:40:38.685843 1443 consensus_queue.cc:237] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } }
I20260924 11:40:38.689658 962 catalog_manager.cc:5495] T c393aa20adb34e088b9c3a343d4ec15d P 68ff49622e73492aa5da588a1707e7b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ff49622e73492aa5da588a1707e7b4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "68ff49622e73492aa5da588a1707e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5d94a7624994956bc47372d22d3d920" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ff49622e73492aa5da588a1707e7b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fd6fd7800a54dcb94f6d431265e860d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42747 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:38.740645 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.727955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.0.97.1:42747)
0924 11:40:38.728187 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.728196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.728474 (+ 278us) server_negotiation.cc:407] Connection header received
0924 11:40:38.728586 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.728589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.728640 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.728711 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.730037 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.730740 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.731963 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.732418 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.732552 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.736214 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.736224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.736236 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.736265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.738497 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.739070 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.739075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.739078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.739199 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.739475 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.739478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.739480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.739746 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.739925 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.740135 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.740309 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:38.780987 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.769072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.0.97.1:42747)
0924 11:40:38.769284 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.769290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.769527 (+ 237us) server_negotiation.cc:407] Connection header received
0924 11:40:38.769651 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.769654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.769710 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.769786 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.770923 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.771607 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.772826 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.773206 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.773293 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.776827 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.776831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.776834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.776861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.779031 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.779535 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.779538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.779540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.779638 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.779907 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.779909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.779911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.780113 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.780325 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.780522 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.780644 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:38.819263 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.807287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.97.1:42747)
0924 11:40:38.807485 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.807492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.807728 (+ 236us) server_negotiation.cc:407] Connection header received
0924 11:40:38.807911 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.807920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.807981 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.808090 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.809260 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.809914 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.811063 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.811441 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.811556 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.815301 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.815306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.815309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.815335 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.817454 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.817946 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.817949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.817951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.818052 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.818307 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.818310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.818312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.818541 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.818703 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.818880 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.819027 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:40:38.820794 1066 tablet_service.cc:2908] Rejecting scan request for tablet c393aa20adb34e088b9c3a343d4ec15d: Uninitialized: clean time has not yet been initialized
I20260924 11:40:38.823526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1023
I20260924 11:40:38.834247 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.824287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.0.97.1:42747)
0924 11:40:38.824416 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.824419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.824783 (+ 364us) server_negotiation.cc:407] Connection header received
0924 11:40:38.824880 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.824883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.824927 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.825003 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.825976 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.826590 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.827668 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.828037 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.828116 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.830128 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.830132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.830135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.830171 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.832500 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.833002 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.833005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.833008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.833101 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.833367 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.833370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.833372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.833581 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.833752 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.833916 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.834053 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:40:38.834770 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:38.918488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1023
I20260924 11:40:38.935178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1157
Sep 24 11:40:38 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:40:38.947922 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:38.935826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.97.2:40423)
0924 11:40:38.936270 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:38.936274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:38.936289 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:40:38.936338 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:38.936342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:38.936398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:38.936484 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:38.937533 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.938207 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.939357 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:38.939730 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:38.939810 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:38.943554 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:38.943564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:38.943578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:38.943603 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:38.945894 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.946457 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.946463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.946465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.946579 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:38.946855 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:38.946858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:38.946860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:38.947173 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:38.947381 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:38.947553 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 11:40:38.947683 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260924 11:40:38.948529 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:39.035087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1157
I20260924 11:40:39.050720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1290
Sep 24 11:40:39 dist-test-slave-qhrh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:40:39.064018 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:39.051366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.0.97.3:36025)
0924 11:40:39.051563 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:39.051567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:39.051584 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:40:39.051723 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:39.051726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:39.051794 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:39.051873 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:39.053087 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:39.053858 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:39.055033 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:39.055391 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:39.055472 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:39.059386 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:39.059396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:39.059410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:39.059441 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:39.061874 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:39.062443 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:39.062449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:39.062452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:39.062561 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:39.062842 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:39.062845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:39.062847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:39.063154 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:39.063337 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:39.063529 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 11:40:39.063663 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:40:39.064623 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:39.145578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1290
I20260924 11:40:39.160851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 930
I20260924 11:40:39.161859 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:39.259519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 930
2026-09-24T11:40:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7906 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:39 dist-test-slave-qhrh krb5kdc[1470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:39 dist-test-slave-qhrh krb5kdc[1470](info): set up 2 sockets
Sep 24 11:40:39 dist-test-slave-qhrh krb5kdc[1470](info): commencing operation
krb5kdc: starting...
W20260924 11:40:40.288021 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:40:40 dist-test-slave-qhrh krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:40:40.443342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:40.747376 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:40.747668 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:40.747797 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:40.754292 1483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:40.754403 1483 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:40:40.754504 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:40.754623 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:40.754702 1483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:40.754783 1483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:40.763209 1483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:40.764264 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:40.765396 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:40.776664 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:40.776695 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:40.776752 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:40.777627 1483 server_base.cc:1034] running on GCE node
I20260924 11:40:40.778288 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:40.779433 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:40.780658 1483 hybrid_clock.cc:648] HybridClock initialized: now 1790250040780607 us; error 58 us; skew 500 ppm
I20260924 11:40:40.780947 1483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:40 dist-test-slave-qhrh krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:40.783739 1483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:40.785008 1483 webserver.cc:460] Webserver started at http://127.0.97.62:36521/ using document root <none> and password file <none>
I20260924 11:40:40.785635 1483 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:40.785759 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:40.786044 1483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:40.788659 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "2f8024d2791a47f086ae4f716179cd10"
format_stamp: "Formatted at 2026-09-24 11:40:40 on dist-test-slave-qhrh"
server_key: "1e01674b9201e1728eaab68f79805968"
server_key_iv: "fccfb2eccd4fe23fe3caaac90293329d"
server_key_version: "encryptionkey@0"
I20260924 11:40:40.789304 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f8024d2791a47f086ae4f716179cd10"
format_stamp: "Formatted at 2026-09-24 11:40:40 on dist-test-slave-qhrh"
server_key: "1e01674b9201e1728eaab68f79805968"
server_key_iv: "fccfb2eccd4fe23fe3caaac90293329d"
server_key_version: "encryptionkey@0"
I20260924 11:40:40.793511 1483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:40:40.796607 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:40.797662 1483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 11:40:40.797837 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "2f8024d2791a47f086ae4f716179cd10"
format_stamp: "Formatted at 2026-09-24 11:40:40 on dist-test-slave-qhrh"
server_key: "1e01674b9201e1728eaab68f79805968"
server_key_iv: "fccfb2eccd4fe23fe3caaac90293329d"
server_key_version: "encryptionkey@0"
I20260924 11:40:40.798030 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:40.823499 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:40.826965 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:40.827203 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:40.846573 1483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45229
I20260924 11:40:40.846591 1559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45229 every 8 connection(s)
I20260924 11:40:40.847716 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:40.850709 1560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:40.856540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1483
I20260924 11:40:40.856751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:40.857509 388 external_mini_cluster.cc:1428] Setting key 342b4d61b82bcb58a4809ca553aa7342
I20260924 11:40:40.860806 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: Bootstrap starting.
I20260924 11:40:40.863420 1560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:40.864466 1560 log.cc:824] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:40.867118 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: No bootstrap required, opened a new log
Sep 24 11:40:40 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:40.876829 1560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8024d2791a47f086ae4f716179cd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45229 } }
I20260924 11:40:40.877131 1560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:40.877242 1560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8024d2791a47f086ae4f716179cd10, State: Initialized, Role: FOLLOWER
I20260924 11:40:40.877825 1560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8024d2791a47f086ae4f716179cd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45229 } }
I20260924 11:40:40.878062 1560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:40.878211 1560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:40.878389 1560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:40.879390 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:40.859353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.97.62:45229)
0924 11:40:40.860037 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:40.860046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:40.860080 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:40:40.861603 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:40.861621 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:40.861926 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:40.862254 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:40.863635 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:40.865003 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:40.866693 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:40.867347 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:40.867478 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:40.872498 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:40.872513 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:40.872530 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:40.872572 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:40.875397 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:40.876130 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:40.876135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:40.876181 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:40.876336 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:40.876727 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:40.876730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:40.876732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:40.877208 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:40.877472 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:40.877923 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 11:40:40.878460 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":345,"threads_started":1}
I20260924 11:40:40.880950 1560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8024d2791a47f086ae4f716179cd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45229 } }
I20260924 11:40:40.881404 1560 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f8024d2791a47f086ae4f716179cd10; no voters:
I20260924 11:40:40.881944 1560 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:40.882066 1565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:40.882827 1565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [term 1 LEADER]: Becoming Leader. State: Replica: 2f8024d2791a47f086ae4f716179cd10, State: Running, Role: LEADER
I20260924 11:40:40.883343 1565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8024d2791a47f086ae4f716179cd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45229 } }
I20260924 11:40:40.883848 1560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:40.886124 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8024d2791a47f086ae4f716179cd10. Latest consensus state: current_term: 1 leader_uuid: "2f8024d2791a47f086ae4f716179cd10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8024d2791a47f086ae4f716179cd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45229 } } }
I20260924 11:40:40.886619 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:40.886113 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f8024d2791a47f086ae4f716179cd10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8024d2791a47f086ae4f716179cd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45229 } } }
I20260924 11:40:40.888181 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:40.888207 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:40.892520 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:40.899847 1570 catalog_manager.cc:1347] Generated new cluster ID: 81bd574cfc6c4bb0946fa230c3a5fd44
I20260924 11:40:40.899933 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:40.913338 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:40.914528 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:40.920017 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: Generated new TSK 0
I20260924 11:40:40.920863 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:40:40.964056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45229
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:41.258665 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:41.258919 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:41.259070 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:41.265699 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:41.265843 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:41.265973 1586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:40:41.266103 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:40:41.274871 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:41.275888 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:41.277138 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:41.289147 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:41.289330 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:41.289573 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:41.289908 1586 server_base.cc:1034] running on GCE node
I20260924 11:40:41.290535 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:41.291316 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:41.292546 1586 hybrid_clock.cc:648] HybridClock initialized: now 1790250041292438 us; error 47 us; skew 500 ppm
I20260924 11:40:41.292810 1586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:41 dist-test-slave-qhrh krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:41.295847 1586 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:40:41.297165 1586 webserver.cc:460] Webserver started at http://127.0.97.1:35041/ using document root <none> and password file <none>
I20260924 11:40:41.297765 1586 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:41.297881 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:41.298182 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:41.300664 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "205b9030a5af45dbaee64c7d58d97961"
format_stamp: "Formatted at 2026-09-24 11:40:41 on dist-test-slave-qhrh"
server_key: "e6d49613736ffb843996d5064cd40ae5"
server_key_iv: "2b2a88d8a14e488ab30db754bc9f7b92"
server_key_version: "encryptionkey@0"
I20260924 11:40:41.301437 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "205b9030a5af45dbaee64c7d58d97961"
format_stamp: "Formatted at 2026-09-24 11:40:41 on dist-test-slave-qhrh"
server_key: "e6d49613736ffb843996d5064cd40ae5"
server_key_iv: "2b2a88d8a14e488ab30db754bc9f7b92"
server_key_version: "encryptionkey@0"
I20260924 11:40:41.305850 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:40:41.308823 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:41.309957 1586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:40:41.310102 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "205b9030a5af45dbaee64c7d58d97961"
format_stamp: "Formatted at 2026-09-24 11:40:41 on dist-test-slave-qhrh"
server_key: "e6d49613736ffb843996d5064cd40ae5"
server_key_iv: "2b2a88d8a14e488ab30db754bc9f7b92"
server_key_version: "encryptionkey@0"
I20260924 11:40:41.310321 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:41.340966 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:41.344928 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:41.345170 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:41.347463 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:41.347591 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:41.347744 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:41.347820 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:40:41 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:41.378640 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.348220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59283 (local address 127.0.97.62:45229)
0924 11:40:41.348392 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:41.348397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:41.349702 (+ 1305us) server_negotiation.cc:407] Connection header received
0924 11:40:41.351569 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:41.351575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:41.351652 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:41.351775 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:41.353653 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.354637 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.356918 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.357658 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.357807 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.371801 (+ 13994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:41.371811 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:41.371814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:41.371841 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:41.375023 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.375946 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.375953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.375957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.376100 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.376723 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.376731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.376737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.377134 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:41.377300 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:41.377885 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 11:40:41.378167 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:40:41.379343 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.348717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.97.1:59283)
0924 11:40:41.349522 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:41.349567 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:41.351255 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:41.351994 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:41.352006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:41.352707 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:41.353483 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.353501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.354777 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.354781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.356754 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.356771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.358165 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.358177 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.358492 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.368626 (+ 10134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:41.368645 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:41.371588 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:41.375146 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.375158 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.375175 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.375765 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.376271 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.376281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.376289 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.376568 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.377275 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:41.377280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:41.377537 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 11:40:41.377727 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 11:40:41.378090 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":385,"threads_started":1}
I20260924 11:40:41.393263 1586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35765
I20260924 11:40:41.393288 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35765 every 8 connection(s)
I20260924 11:40:41.394570 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:40:41.396394 1721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45229
I20260924 11:40:41.396646 1721 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:41.397370 1721 heartbeater.cc:511] Master 127.0.97.62:45229 requested a full tablet report, sending...
I20260924 11:40:41.397998 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.384689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43911 (local address 127.0.97.62:45229)
0924 11:40:41.384915 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:41.384923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:41.385916 (+ 993us) server_negotiation.cc:407] Connection header received
0924 11:40:41.386038 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:41.386045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:41.386132 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:41.386278 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:41.387028 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.388402 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.390680 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.391100 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.391223 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.392974 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:41.392980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:41.392985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:41.393016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:41.395366 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.396344 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.396352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.396355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.396480 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.396909 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.396912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.396914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.397134 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:41.397232 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:41.397640 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 11:40:41.397783 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 11:40:41.398432 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.385101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.97.1:43911)
0924 11:40:41.385821 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:41.385841 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:41.385933 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:41.386299 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:41.386302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:41.386645 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:41.386898 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.386905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.388571 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.388579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.390552 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.390582 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.391275 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.391286 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.391457 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.392020 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:41.392034 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:41.392805 (+ 771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:41.395607 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.395611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.395615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.396227 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.396600 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.396604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.396609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.396811 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.397252 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:41.397257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:41.397426 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 11:40:41.397578 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 11:40:41.397848 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":419,"threads_started":1}
I20260924 11:40:41.398874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1586
I20260924 11:40:41.399056 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:40:41.399438 1515 ts_manager.cc:194] Registered new tserver with Master: 205b9030a5af45dbaee64c7d58d97961 (127.0.97.1:35765)
I20260924 11:40:41.399775 388 external_mini_cluster.cc:1428] Setting key ccfebc395945d1ae13bcff2c66fe20cf
I20260924 11:40:41.401316 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59283
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:40:41.438651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45229
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:41.722409 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:41.722661 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:41.722807 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:41.729586 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:41.729671 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:41.729764 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:40:41.729908 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:40:41.738592 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:41.739639 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:41.740856 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:41.752194 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:41.752251 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:41.752233 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:41.753122 1725 server_base.cc:1034] running on GCE node
I20260924 11:40:41.753650 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:41.754411 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:41.755601 1725 hybrid_clock.cc:648] HybridClock initialized: now 1790250041755555 us; error 30 us; skew 500 ppm
I20260924 11:40:41.755896 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:41 dist-test-slave-qhrh krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:41.758766 1725 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:40:41.759981 1725 webserver.cc:460] Webserver started at http://127.0.97.2:44605/ using document root <none> and password file <none>
I20260924 11:40:41.760643 1725 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:41.760751 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:41.761034 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:41.763619 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "85a5f2cc79b74d5b9aefec2c215e466d"
format_stamp: "Formatted at 2026-09-24 11:40:41 on dist-test-slave-qhrh"
server_key: "ce521b62fba811ee70c47fef22459e16"
server_key_iv: "86e12cbead321ffd736a06afd1251f16"
server_key_version: "encryptionkey@0"
I20260924 11:40:41.764398 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "85a5f2cc79b74d5b9aefec2c215e466d"
format_stamp: "Formatted at 2026-09-24 11:40:41 on dist-test-slave-qhrh"
server_key: "ce521b62fba811ee70c47fef22459e16"
server_key_iv: "86e12cbead321ffd736a06afd1251f16"
server_key_version: "encryptionkey@0"
I20260924 11:40:41.768819 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:40:41.771661 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:41.772852 1725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:40:41.773006 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "85a5f2cc79b74d5b9aefec2c215e466d"
format_stamp: "Formatted at 2026-09-24 11:40:41 on dist-test-slave-qhrh"
server_key: "ce521b62fba811ee70c47fef22459e16"
server_key_iv: "86e12cbead321ffd736a06afd1251f16"
server_key_version: "encryptionkey@0"
I20260924 11:40:41.773202 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:41.794906 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:41.798748 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:41.798995 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:41.801390 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:41.801484 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:41.801652 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:41.801748 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 11:40:41 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:41.824221 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.801783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36347 (local address 127.0.97.62:45229)
0924 11:40:41.801967 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:41.801973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:41.803317 (+ 1344us) server_negotiation.cc:407] Connection header received
0924 11:40:41.805564 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:41.805572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:41.805671 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:41.805787 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:41.807853 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.808902 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.811801 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.812460 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.812615 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.816971 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:41.816981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:41.816988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:41.817040 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:41.820516 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.821454 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.821462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.821468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.821611 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.822174 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.822182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.822188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.822647 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:41.822815 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:41.823396 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 11:40:41.823698 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:40:41.824846 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.802337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.97.2:36347)
0924 11:40:41.803150 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:41.803198 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:41.805254 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:41.806035 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:41.806042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:41.806764 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:41.807621 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.807647 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.809095 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.809103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.811644 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.811661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.812743 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.812754 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.813083 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.813846 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:41.813881 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:41.816740 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:41.820637 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.820644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.820659 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.821240 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.821740 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.821743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.821745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.822036 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.822787 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:41.822792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:41.823057 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 11:40:41.823263 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 11:40:41.823573 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":377,"threads_started":1}
I20260924 11:40:41.844058 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.829916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49427 (local address 127.0.97.62:45229)
0924 11:40:41.830215 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:41.830222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:41.830730 (+ 508us) server_negotiation.cc:407] Connection header received
0924 11:40:41.830972 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:41.830978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:41.831066 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:41.831139 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:41.832677 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.833562 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.835534 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.836101 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.836276 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.838416 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:41.838420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:41.838423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:41.838447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:41.841294 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.842157 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.842161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.842164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.842320 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:41.842792 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:41.842795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:41.842797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:41.843060 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:41.843171 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:41.843704 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 11:40:41.843845 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260924 11:40:41.844693 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:41.829917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.97.2:49427)
0924 11:40:41.830610 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:41.830627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:41.830822 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:41.831320 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:41.831322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:41.831963 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:41.832461 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.832476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.833697 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.833700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.835380 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:41.835395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:41.836344 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:41.836353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:41.836505 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:41.837170 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:41.837184 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:41.838218 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:41.841410 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.841416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.841420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.842016 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.842437 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:41.842446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:41.842450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:41.842662 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:41.843175 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:41.843183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:41.843417 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 11:40:41.843612 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 11:40:41.844026 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":471,"threads_started":1}
I20260924 11:40:41.853190 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39655
I20260924 11:40:41.853266 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39655 every 8 connection(s)
I20260924 11:40:41.854238 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:40:41.854632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1725
I20260924 11:40:41.854796 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:40:41.855428 388 external_mini_cluster.cc:1428] Setting key e4783148d1823bc45aee55c5086fb43c
I20260924 11:40:41.855942 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45229
I20260924 11:40:41.856135 1860 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:41.856849 1860 heartbeater.cc:511] Master 127.0.97.62:45229 requested a full tablet report, sending...
I20260924 11:40:41.858098 1515 ts_manager.cc:194] Registered new tserver with Master: 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655)
I20260924 11:40:41.859069 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36347
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:40:41.892550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45229
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:42.183233 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:42.183476 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:42.183562 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:42.190408 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:42.190516 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:42.190591 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:40:42.190716 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:40:42.199319 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:42.200399 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:42.201646 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:42.213559 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:42.213642 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:42.213673 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:42.214502 1864 server_base.cc:1034] running on GCE node
I20260924 11:40:42.215132 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:42.215955 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:42.217188 1864 hybrid_clock.cc:648] HybridClock initialized: now 1790250042217128 us; error 56 us; skew 500 ppm
I20260924 11:40:42.217459 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:42 dist-test-slave-qhrh krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:42.220860 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:40:42.222244 1864 webserver.cc:460] Webserver started at http://127.0.97.3:37299/ using document root <none> and password file <none>
I20260924 11:40:42.222908 1864 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:42.223040 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:42.223371 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:42.226202 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "54d21b78c5fc45eb8b88bcfa13463110"
format_stamp: "Formatted at 2026-09-24 11:40:42 on dist-test-slave-qhrh"
server_key: "96578de13e0a4bdffcd97c6320cdca65"
server_key_iv: "b9d84faf61a480e1f246db33ec93d83f"
server_key_version: "encryptionkey@0"
I20260924 11:40:42.227063 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "54d21b78c5fc45eb8b88bcfa13463110"
format_stamp: "Formatted at 2026-09-24 11:40:42 on dist-test-slave-qhrh"
server_key: "96578de13e0a4bdffcd97c6320cdca65"
server_key_iv: "b9d84faf61a480e1f246db33ec93d83f"
server_key_version: "encryptionkey@0"
I20260924 11:40:42.231824 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:40:42.235131 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:42.236367 1864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:40:42.236565 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "54d21b78c5fc45eb8b88bcfa13463110"
format_stamp: "Formatted at 2026-09-24 11:40:42 on dist-test-slave-qhrh"
server_key: "96578de13e0a4bdffcd97c6320cdca65"
server_key_iv: "b9d84faf61a480e1f246db33ec93d83f"
server_key_version: "encryptionkey@0"
I20260924 11:40:42.236765 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:42.250960 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:42.254460 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:42.254715 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:42.257302 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:42.257400 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:42.257534 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:42.257622 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:40:42 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:42.279403 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.257823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.97.3:45655)
0924 11:40:42.258597 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.258635 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.260312 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.261065 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.261077 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.261533 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:42.262320 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.262344 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.263664 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.263669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.265784 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.265798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.266782 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.266790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.267114 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.267810 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:42.267845 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:42.270982 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:42.274879 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:42.274889 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:42.274906 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:42.275642 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:42.276034 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:42.276043 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:42.276049 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:42.276386 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:42.277084 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:42.277092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:42.277455 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.277773 (+ 318us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.278195 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":379,"threads_started":1}
I20260924 11:40:42.280565 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.257409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45655 (local address 127.0.97.62:45229)
0924 11:40:42.257605 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.257611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.258751 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 11:40:42.260570 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.260576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.260662 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.260801 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:42.262485 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.263515 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.265932 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.266542 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.266676 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.271287 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:42.271297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:42.271305 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:42.271356 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:42.274750 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:42.275778 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:42.275785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:42.275790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:42.275934 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:42.276546 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:42.276553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:42.276558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:42.276847 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:42.276998 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.280080 (+ 3082us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.280293 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:40:42.299687 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.285335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43767 (local address 127.0.97.62:45229)
0924 11:40:42.285554 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.285562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.285986 (+ 424us) server_negotiation.cc:407] Connection header received
0924 11:40:42.286109 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.286118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.286215 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.286367 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:42.287090 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.288463 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.290446 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.291032 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.291178 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.293957 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:42.293963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:42.293968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:42.294010 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:42.296651 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:42.297687 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:42.297695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:42.297701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:42.297884 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:42.298325 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:42.298330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:42.298333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:42.298667 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:42.298802 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.299168 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.299350 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:40:42.299775 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.285328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.97.3:43767)
0924 11:40:42.285891 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.285915 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.285999 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.286367 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.286370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.286717 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:42.286960 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.286967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.288612 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.288618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.290319 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.290330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.291269 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.291278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.291536 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.292182 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:42.292221 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:42.293772 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:42.296775 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:42.296798 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:42.296802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:42.297528 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:42.298017 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:42.298024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:42.298041 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:42.298189 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:42.298773 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:42.298778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:42.298945 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.299084 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.299351 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":315,"threads_started":1}
I20260924 11:40:42.309038 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39035
I20260924 11:40:42.309054 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39035 every 8 connection(s)
I20260924 11:40:42.310477 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:40:42.312244 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45229
I20260924 11:40:42.312438 1999 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:42.313022 1999 heartbeater.cc:511] Master 127.0.97.62:45229 requested a full tablet report, sending...
I20260924 11:40:42.314414 1515 ts_manager.cc:194] Registered new tserver with Master: 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035)
I20260924 11:40:42.315124 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45655
I20260924 11:40:42.315627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 1864
I20260924 11:40:42.315789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:40:42.316298 388 external_mini_cluster.cc:1428] Setting key bc7da7cb142061f5d6f356490ae7e04f
I20260924 11:40:42.318848 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:40:42 dist-test-slave-qhrh krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:40:42 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:42.358687 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.347149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.97.62:45229)
0924 11:40:42.347300 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.347304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.347619 (+ 315us) server_negotiation.cc:407] Connection header received
0924 11:40:42.347749 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.347752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.347806 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.347883 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:42.349153 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.349763 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.350814 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.351191 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.351272 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.354669 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:42.354674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:42.354677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:42.354706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:42.356876 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:42.357384 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:42.357387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:42.357389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:42.357486 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:42.357763 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:42.357766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:42.357768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:42.357974 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:42.358143 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.358342 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.358479 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:40:42.362197 1515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:40:42.368762 1515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:42.392799 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.379576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39035 (local address 127.0.0.1:53696)
0924 11:40:42.380853 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.380951 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.381212 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.383045 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.383055 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.383100 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:42.383537 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.383551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.386255 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.386264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.389301 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.389322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.391913 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.391919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.392086 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.392129 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.392264 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.392418 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":809,"threads_started":1}
I20260924 11:40:42.393569 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.379804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.0.97.3:39035)
0924 11:40:42.382208 (+ 2404us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.382216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.382244 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:40:42.382350 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.382360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.382640 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.382848 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:42.383961 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.386033 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.389608 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.391450 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.391590 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.391806 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.393124 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.393273 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2242,"thread_start_us":2029,"threads_started":1}
I20260924 11:40:42.397810 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.379790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35765 (local address 127.0.0.1:58416)
0924 11:40:42.381404 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.381421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.381562 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.384175 (+ 2613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.384183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.384215 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:42.384725 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.384738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.389069 (+ 4331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.389076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.393018 (+ 3942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.393036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.396700 (+ 3664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.396708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.396956 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.397003 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.397076 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.397248 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1475,"thread_start_us":1346,"threads_started":1}
I20260924 11:40:42.397723 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 83155369e5664b5185df2089e47fffff (DEFAULT_TABLE table=test-table [id=6e84f30b52fb42d2b61c8d517f79980c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:42.399009 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.380263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.0.97.1:35765)
0924 11:40:42.382364 (+ 2101us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.382372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.382397 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:40:42.383659 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.383671 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.383926 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.384937 (+ 1011us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:42.385711 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.388896 (+ 3185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.393278 (+ 4382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.395269 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.398025 (+ 2756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.398267 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.398358 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.398589 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":2086,"threads_started":1}
I20260924 11:40:42.399147 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83155369e5664b5185df2089e47fffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:42.402691 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 83155369e5664b5185df2089e47fffff (DEFAULT_TABLE table=test-table [id=6e84f30b52fb42d2b61c8d517f79980c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:42.403888 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83155369e5664b5185df2089e47fffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:42.405251 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.382471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.0.97.2:39655)
0924 11:40:42.399546 (+ 17075us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.399555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.399584 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:40:42.399691 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.399698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.399909 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.400156 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:42.400430 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.401659 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.403407 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.404356 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.404480 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.404658 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.404864 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.405000 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16933,"thread_start_us":16830,"threads_started":1}
I20260924 11:40:42.405264 1721 heartbeater.cc:503] Master 127.0.97.62:45229 was elected leader, sending a full tablet report...
I20260924 11:40:42.405915 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.382244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39655 (local address 127.0.0.1:44924)
0924 11:40:42.394181 (+ 11937us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.394199 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.394281 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.400156 (+ 5875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.400160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.400185 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:42.400323 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.400335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.402009 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.402014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.403218 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.403235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.404503 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.404509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.404672 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.404704 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.404822 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.404939 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11815,"thread_start_us":17329,"threads_started":1}
I20260924 11:40:42.408854 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 83155369e5664b5185df2089e47fffff (DEFAULT_TABLE table=test-table [id=6e84f30b52fb42d2b61c8d517f79980c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:42.409806 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83155369e5664b5185df2089e47fffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:42.413340 2018 tablet_bootstrap.cc:492] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Bootstrap starting.
I20260924 11:40:42.416567 2018 tablet_bootstrap.cc:654] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:42.417153 2019 tablet_bootstrap.cc:492] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Bootstrap starting.
I20260924 11:40:42.417689 2018 log.cc:824] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:42.419862 2019 tablet_bootstrap.cc:654] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:42.420436 2018 tablet_bootstrap.cc:492] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: No bootstrap required, opened a new log
I20260924 11:40:42.420701 2018 ts_tablet_manager.cc:1397] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 11:40:42.421008 2019 log.cc:824] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:42.423416 2019 tablet_bootstrap.cc:492] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: No bootstrap required, opened a new log
I20260924 11:40:42.423647 2019 ts_tablet_manager.cc:1397] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 11:40:42.424214 2021 tablet_bootstrap.cc:492] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Bootstrap starting.
I20260924 11:40:42.427716 2021 tablet_bootstrap.cc:654] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:42.428486 2018 raft_consensus.cc:348] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.428764 2021 log.cc:824] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:42.428786 2018 raft_consensus.cc:374] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:42.428896 2018 raft_consensus.cc:729] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d21b78c5fc45eb8b88bcfa13463110, State: Initialized, Role: FOLLOWER
I20260924 11:40:42.429417 2018 consensus_queue.cc:260] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.433446 2018 ts_tablet_manager.cc:1428] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 11:40:42.433581 1999 heartbeater.cc:503] Master 127.0.97.62:45229 was elected leader, sending a full tablet report...
I20260924 11:40:42.434154 2019 raft_consensus.cc:348] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.434568 2019 raft_consensus.cc:374] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:42.434669 2019 raft_consensus.cc:729] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205b9030a5af45dbaee64c7d58d97961, State: Initialized, Role: FOLLOWER
I20260924 11:40:42.434892 2021 tablet_bootstrap.cc:492] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: No bootstrap required, opened a new log
I20260924 11:40:42.435114 2021 ts_tablet_manager.cc:1397] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260924 11:40:42.435213 2019 consensus_queue.cc:260] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.437130 2019 ts_tablet_manager.cc:1428] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260924 11:40:42.442905 2021 raft_consensus.cc:348] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.443215 2021 raft_consensus.cc:374] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:42.443323 2021 raft_consensus.cc:729] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a5f2cc79b74d5b9aefec2c215e466d, State: Initialized, Role: FOLLOWER
I20260924 11:40:42.443854 2021 consensus_queue.cc:260] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.444942 1860 heartbeater.cc:503] Master 127.0.97.62:45229 was elected leader, sending a full tablet report...
I20260924 11:40:42.445400 2021 ts_tablet_manager.cc:1428] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
W20260924 11:40:42.561973 2000 tablet.cc:2369] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:40:42.614282 1861 tablet.cc:2369] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:40:42.650311 1722 tablet.cc:2369] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:40:42.712040 2025 raft_consensus.cc:482] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:42.712777 2025 raft_consensus.cc:504] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.714406 2025 leader_election.cc:288] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035), 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655)
I20260924 11:40:42.720558 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.714972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45957 (local address 127.0.97.2:39655)
0924 11:40:42.715193 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.715202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.715749 (+ 547us) server_negotiation.cc:407] Connection header received
0924 11:40:42.715945 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.715950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.716031 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.716144 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:42.716628 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.717295 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.718901 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.719605 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.719747 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.719845 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.720181 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.720293 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:40:42.720676 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.714826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39053 (local address 127.0.97.3:39035)
0924 11:40:42.715052 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.715062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.715884 (+ 822us) server_negotiation.cc:407] Connection header received
0924 11:40:42.716092 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.716096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.716162 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.716288 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:42.716824 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.717469 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.719023 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.719737 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.719876 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.719978 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.720313 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.720429 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:40:42.720700 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.714691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39655 (local address 127.0.97.1:45957)
0924 11:40:42.715653 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.715667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.715822 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.716182 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.716186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.716206 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:42.716505 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.716513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.717411 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.717417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.718761 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.718771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.719792 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.719797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.719944 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.720038 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.720141 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.720270 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":42,"spinlock_wait_cycles":22784,"thread_start_us":811,"threads_started":1}
I20260924 11:40:42.720852 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.714816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39035 (local address 127.0.97.1:39053)
0924 11:40:42.715786 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:42.715798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:42.715928 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:42.716307 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:42.716314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:42.716355 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:42.716686 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.716693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.717605 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.717610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.718868 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:42.718879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.719936 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.719942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:42.720121 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.720189 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 11:40:42.720291 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:40:42.720405 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":31,"spinlock_wait_cycles":768,"thread_start_us":403,"threads_started":1}
I20260924 11:40:42.721684 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83155369e5664b5185df2089e47fffff" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" is_pre_election: true
I20260924 11:40:42.721968 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83155369e5664b5185df2089e47fffff" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d21b78c5fc45eb8b88bcfa13463110" is_pre_election: true
I20260924 11:40:42.722188 1815 raft_consensus.cc:2454] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 0.
I20260924 11:40:42.722955 1954 raft_consensus.cc:2454] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 0.
I20260924 11:40:42.723258 1602 leader_election.cc:302] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205b9030a5af45dbaee64c7d58d97961, 85a5f2cc79b74d5b9aefec2c215e466d; no voters:
I20260924 11:40:42.723661 2025 raft_consensus.cc:2781] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:42.723788 2025 raft_consensus.cc:482] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:42.723891 2025 raft_consensus.cc:3037] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:42.725888 2025 raft_consensus.cc:504] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.726457 2025 leader_election.cc:288] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 election: Requested vote from peers 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035), 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655)
I20260924 11:40:42.726783 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83155369e5664b5185df2089e47fffff" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d21b78c5fc45eb8b88bcfa13463110"
I20260924 11:40:42.726907 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83155369e5664b5185df2089e47fffff" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a5f2cc79b74d5b9aefec2c215e466d"
I20260924 11:40:42.726965 1954 raft_consensus.cc:3037] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:42.727088 1815 raft_consensus.cc:3037] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:42.730384 1954 raft_consensus.cc:2454] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 1.
I20260924 11:40:42.730516 1815 raft_consensus.cc:2454] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 1.
I20260924 11:40:42.730775 1604 leader_election.cc:302] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205b9030a5af45dbaee64c7d58d97961, 54d21b78c5fc45eb8b88bcfa13463110; no voters:
I20260924 11:40:42.731045 2025 raft_consensus.cc:2781] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:42.731418 2025 raft_consensus.cc:686] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 1 LEADER]: Becoming Leader. State: Replica: 205b9030a5af45dbaee64c7d58d97961, State: Running, Role: LEADER
I20260924 11:40:42.731817 2025 consensus_queue.cc:237] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } }
I20260924 11:40:42.735716 1515 catalog_manager.cc:5495] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 205b9030a5af45dbaee64c7d58d97961 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } health_report { overall_health: UNKNOWN } } }
I20260924 11:40:42.757753 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:42.752772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.97.1:35765)
0924 11:40:42.752968 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:42.752976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:42.753130 (+ 154us) server_negotiation.cc:407] Connection header received
0924 11:40:42.753425 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:42.753432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:42.753517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:42.753652 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:42.753993 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.754657 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.755815 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:42.756231 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:42.756348 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:42.756766 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:42.757172 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:42.757320 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:42.757443 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:40:42.757518 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:40:42.764343 1815 raft_consensus.cc:1261] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Refusing update from remote peer 205b9030a5af45dbaee64c7d58d97961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:42.764376 1954 raft_consensus.cc:1261] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Refusing update from remote peer 205b9030a5af45dbaee64c7d58d97961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:42.765185 2029 consensus_queue.cc:1035] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:42.765513 2025 consensus_queue.cc:1035] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:42.775507 2032 mvcc.cc:205] Tried to move back new op lower bound from 7332864175154475008 to 7332864175033040896. Current Snapshot: MvccSnapshot[applied={T|T < 7332864175154475008}]
I20260924 11:40:42.795342 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:42.795567 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:42.798643 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: Sending DeleteTablet for 3 replicas of tablet 83155369e5664b5185df2089e47fffff
I20260924 11:40:42.799554 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 83155369e5664b5185df2089e47fffff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53696
I20260924 11:40:42.799638 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 83155369e5664b5185df2089e47fffff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44924
I20260924 11:40:42.799962 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 83155369e5664b5185df2089e47fffff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58416
I20260924 11:40:42.800786 2043 tablet_replica.cc:330] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: stopping tablet replica
I20260924 11:40:42.801129 2043 raft_consensus.cc:2229] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:42.801270 1515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 11:40:42.801604 2043 raft_consensus.cc:2258] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 11:40:42.801996 1515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:42.806217 2043 ts_tablet_manager.cc:1905] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:42.810199 1650 tablet_service.cc:1467] Processing CreateTablet for tablet dc50fe5ea53d416e89d1ca28d377dbd9 (DEFAULT_TABLE table=test-table [id=8ee49c7100a94d78bc45d9987ebc599a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:42.810526 1928 tablet_service.cc:1467] Processing CreateTablet for tablet dc50fe5ea53d416e89d1ca28d377dbd9 (DEFAULT_TABLE table=test-table [id=8ee49c7100a94d78bc45d9987ebc599a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:42.810791 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc50fe5ea53d416e89d1ca28d377dbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:42.810906 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc50fe5ea53d416e89d1ca28d377dbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:42.812438 1790 tablet_service.cc:1467] Processing CreateTablet for tablet dc50fe5ea53d416e89d1ca28d377dbd9 (DEFAULT_TABLE table=test-table [id=8ee49c7100a94d78bc45d9987ebc599a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:42.812819 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc50fe5ea53d416e89d1ca28d377dbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:42.818850 2018 tablet_bootstrap.cc:492] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Bootstrap starting.
I20260924 11:40:42.820319 2018 tablet_bootstrap.cc:654] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:42.823144 2021 tablet_bootstrap.cc:492] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Bootstrap starting.
I20260924 11:40:42.824784 2021 tablet_bootstrap.cc:654] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:42.829476 2019 tablet_bootstrap.cc:492] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Bootstrap starting.
I20260924 11:40:42.831357 2044 tablet_replica.cc:330] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: stopping tablet replica
I20260924 11:40:42.831456 2019 tablet_bootstrap.cc:654] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:42.831848 2044 raft_consensus.cc:2229] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:40:42.832681 2044 raft_consensus.cc:2258] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:42.839087 2044 ts_tablet_manager.cc:1905] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:42.841284 2018 tablet_bootstrap.cc:492] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: No bootstrap required, opened a new log
I20260924 11:40:42.841464 2018 ts_tablet_manager.cc:1397] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.008s
I20260924 11:40:42.842176 2021 tablet_bootstrap.cc:492] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: No bootstrap required, opened a new log
I20260924 11:40:42.842295 2021 ts_tablet_manager.cc:1397] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.006s
I20260924 11:40:42.842929 2018 raft_consensus.cc:348] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.843071 2021 raft_consensus.cc:348] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.843262 2021 raft_consensus.cc:374] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:42.843222 2018 raft_consensus.cc:374] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:42.843353 2021 raft_consensus.cc:729] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a5f2cc79b74d5b9aefec2c215e466d, State: Initialized, Role: FOLLOWER
I20260924 11:40:42.843468 2018 raft_consensus.cc:729] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d21b78c5fc45eb8b88bcfa13463110, State: Initialized, Role: FOLLOWER
I20260924 11:40:42.843667 2021 consensus_queue.cc:260] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.843770 2018 consensus_queue.cc:260] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.845953 2018 ts_tablet_manager.cc:1428] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 11:40:42.840579 2042 tablet_replica.cc:330] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: stopping tablet replica
I20260924 11:40:42.849917 2042 raft_consensus.cc:2229] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:42.850306 2042 raft_consensus.cc:2258] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:42.850766 2021 ts_tablet_manager.cc:1428] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 11:40:42.852172 2042 ts_tablet_manager.cc:1905] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:42.858310 2042 ts_tablet_manager.cc:1918] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:40:42.858461 2042 log.cc:1196] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/83155369e5664b5185df2089e47fffff
I20260924 11:40:42.858930 2042 ts_tablet_manager.cc:1939] T 83155369e5664b5185df2089e47fffff P 54d21b78c5fc45eb8b88bcfa13463110: Deleting consensus metadata
I20260924 11:40:42.865984 2043 ts_tablet_manager.cc:1918] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:40:42.867235 1501 catalog_manager.cc:4841] TS 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035): tablet 83155369e5664b5185df2089e47fffff (table test-table [id=6e84f30b52fb42d2b61c8d517f79980c]) successfully deleted
I20260924 11:40:42.869676 2043 log.cc:1196] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/83155369e5664b5185df2089e47fffff
I20260924 11:40:42.870325 2043 ts_tablet_manager.cc:1939] T 83155369e5664b5185df2089e47fffff P 85a5f2cc79b74d5b9aefec2c215e466d: Deleting consensus metadata
I20260924 11:40:42.872256 1499 catalog_manager.cc:4841] TS 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655): tablet 83155369e5664b5185df2089e47fffff (table test-table [id=6e84f30b52fb42d2b61c8d517f79980c]) successfully deleted
I20260924 11:40:42.880095 2044 ts_tablet_manager.cc:1918] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:40:42.880312 2044 log.cc:1196] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/83155369e5664b5185df2089e47fffff
I20260924 11:40:42.880879 2044 ts_tablet_manager.cc:1939] T 83155369e5664b5185df2089e47fffff P 205b9030a5af45dbaee64c7d58d97961: Deleting consensus metadata
I20260924 11:40:42.883042 2019 tablet_bootstrap.cc:492] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: No bootstrap required, opened a new log
I20260924 11:40:42.883188 2019 ts_tablet_manager.cc:1397] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.022s
I20260924 11:40:42.883330 1499 catalog_manager.cc:4841] TS 205b9030a5af45dbaee64c7d58d97961 (127.0.97.1:35765): tablet 83155369e5664b5185df2089e47fffff (table test-table [id=6e84f30b52fb42d2b61c8d517f79980c]) successfully deleted
I20260924 11:40:42.884217 2019 raft_consensus.cc:348] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.884433 2019 raft_consensus.cc:374] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:42.884532 2019 raft_consensus.cc:729] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205b9030a5af45dbaee64c7d58d97961, State: Initialized, Role: FOLLOWER
I20260924 11:40:42.884784 2019 consensus_queue.cc:260] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.885308 2019 ts_tablet_manager.cc:1428] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:42.963160 2025 raft_consensus.cc:482] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:42.963351 2025 raft_consensus.cc:504] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.964053 2025 leader_election.cc:288] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655), 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035)
I20260924 11:40:42.964501 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc50fe5ea53d416e89d1ca28d377dbd9" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" is_pre_election: true
I20260924 11:40:42.964541 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc50fe5ea53d416e89d1ca28d377dbd9" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d21b78c5fc45eb8b88bcfa13463110" is_pre_election: true
I20260924 11:40:42.964722 1815 raft_consensus.cc:2454] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 0.
I20260924 11:40:42.964751 1954 raft_consensus.cc:2454] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 0.
I20260924 11:40:42.965196 1604 leader_election.cc:302] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205b9030a5af45dbaee64c7d58d97961, 54d21b78c5fc45eb8b88bcfa13463110; no voters:
I20260924 11:40:42.965497 2025 raft_consensus.cc:2781] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:42.965631 2025 raft_consensus.cc:482] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:42.965754 2025 raft_consensus.cc:3037] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:42.967873 2025 raft_consensus.cc:504] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.968493 2025 leader_election.cc:288] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 election: Requested vote from peers 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655), 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035)
I20260924 11:40:42.968850 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc50fe5ea53d416e89d1ca28d377dbd9" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a5f2cc79b74d5b9aefec2c215e466d"
I20260924 11:40:42.968950 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc50fe5ea53d416e89d1ca28d377dbd9" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d21b78c5fc45eb8b88bcfa13463110"
I20260924 11:40:42.969043 1815 raft_consensus.cc:3037] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:42.969141 1954 raft_consensus.cc:3037] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:42.971765 1954 raft_consensus.cc:2454] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 1.
I20260924 11:40:42.971765 1815 raft_consensus.cc:2454] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 1.
I20260924 11:40:42.972078 1602 leader_election.cc:302] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205b9030a5af45dbaee64c7d58d97961, 85a5f2cc79b74d5b9aefec2c215e466d; no voters:
I20260924 11:40:42.972389 2025 raft_consensus.cc:2781] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:42.972524 2025 raft_consensus.cc:686] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 1 LEADER]: Becoming Leader. State: Replica: 205b9030a5af45dbaee64c7d58d97961, State: Running, Role: LEADER
I20260924 11:40:42.972796 2025 consensus_queue.cc:237] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:42.975494 1513 catalog_manager.cc:5495] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 205b9030a5af45dbaee64c7d58d97961 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:43.060678 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.046575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45229 (local address 127.0.0.1:43428)
0924 11:40:43.047240 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:43.047263 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:43.047394 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:43.048224 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:43.048231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:43.048994 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:43.049254 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:43.049265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.050217 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.050222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:43.051603 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:43.051614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.052227 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.052233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:43.052396 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.053055 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:43.053077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:43.055425 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:43.058265 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:43.058276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:43.058283 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:43.058672 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:43.059025 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:43.059030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:43.059033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:43.059167 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:43.059677 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:43.059683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:43.059814 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 11:40:43.059961 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 11:40:43.060267 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":336,"threads_started":1}
I20260924 11:40:43.060731 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.046575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.97.62:45229)
0924 11:40:43.047586 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.047590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.047990 (+ 400us) server_negotiation.cc:407] Connection header received
0924 11:40:43.048033 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.048035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.048081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.048192 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:43.049385 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.050097 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.051718 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.052062 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.052186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.055611 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:43.055616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:43.055619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:43.055641 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:43.058124 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.058779 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.058787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.058791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.058915 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.059261 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.059268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.059272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.059542 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:43.059688 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.060033 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.060382 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":917,"thread_start_us":775,"threads_started":1}
I20260924 11:40:43.065829 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43428:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:40:43.066828 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:43.073321 1651 tablet_service.cc:1467] Processing CreateTablet for tablet a15fa46813d24fd0ae65157192350321 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21b1780c24f546df9accce325f1eac01]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:40:43.073604 1790 tablet_service.cc:1467] Processing CreateTablet for tablet a15fa46813d24fd0ae65157192350321 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21b1780c24f546df9accce325f1eac01]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:40:43.074107 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15fa46813d24fd0ae65157192350321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:43.074211 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15fa46813d24fd0ae65157192350321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:43.074359 1929 tablet_service.cc:1467] Processing CreateTablet for tablet a15fa46813d24fd0ae65157192350321 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21b1780c24f546df9accce325f1eac01]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:40:43.074864 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15fa46813d24fd0ae65157192350321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:43.082077 2019 tablet_bootstrap.cc:492] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: Bootstrap starting.
I20260924 11:40:43.082458 2021 tablet_bootstrap.cc:492] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: Bootstrap starting.
I20260924 11:40:43.083717 2019 tablet_bootstrap.cc:654] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:43.084086 2021 tablet_bootstrap.cc:654] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:43.085847 2019 tablet_bootstrap.cc:492] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: No bootstrap required, opened a new log
I20260924 11:40:43.086052 2019 ts_tablet_manager.cc:1397] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 11:40:43.086675 2019 raft_consensus.cc:348] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.086884 2019 raft_consensus.cc:374] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:43.086988 2019 raft_consensus.cc:729] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205b9030a5af45dbaee64c7d58d97961, State: Initialized, Role: FOLLOWER
I20260924 11:40:43.087260 2019 consensus_queue.cc:260] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.087680 2025 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.087996 2025 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.088392 2018 tablet_bootstrap.cc:492] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: Bootstrap starting.
I20260924 11:40:43.090392 2018 tablet_bootstrap.cc:654] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:43.091406 2019 ts_tablet_manager.cc:1428] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 11:40:43.093816 2018 tablet_bootstrap.cc:492] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: No bootstrap required, opened a new log
I20260924 11:40:43.093966 2021 tablet_bootstrap.cc:492] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: No bootstrap required, opened a new log
I20260924 11:40:43.094033 2018 ts_tablet_manager.cc:1397] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 11:40:43.094136 2021 ts_tablet_manager.cc:1397] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20260924 11:40:43.094614 2018 raft_consensus.cc:348] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.094810 2018 raft_consensus.cc:374] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:43.094878 2018 raft_consensus.cc:729] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d21b78c5fc45eb8b88bcfa13463110, State: Initialized, Role: FOLLOWER
I20260924 11:40:43.094815 2021 raft_consensus.cc:348] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.095038 2021 raft_consensus.cc:374] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:43.095117 2021 raft_consensus.cc:729] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a5f2cc79b74d5b9aefec2c215e466d, State: Initialized, Role: FOLLOWER
I20260924 11:40:43.095134 2018 consensus_queue.cc:260] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.095330 2021 consensus_queue.cc:260] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.095630 2018 ts_tablet_manager.cc:1428] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 11:40:43.095542 2024 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.095822 2024 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.095715 2026 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.095984 2026 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.097184 2021 ts_tablet_manager.cc:1428] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 11:40:43.140918 2025 raft_consensus.cc:482] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:43.141129 2025 raft_consensus.cc:504] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.141767 2025 leader_election.cc:288] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655), 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035)
I20260924 11:40:43.142170 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15fa46813d24fd0ae65157192350321" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" is_pre_election: true
I20260924 11:40:43.142405 1815 raft_consensus.cc:2454] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 0.
I20260924 11:40:43.142390 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15fa46813d24fd0ae65157192350321" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d21b78c5fc45eb8b88bcfa13463110" is_pre_election: true
I20260924 11:40:43.142611 1954 raft_consensus.cc:2454] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 0.
I20260924 11:40:43.142853 1602 leader_election.cc:302] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205b9030a5af45dbaee64c7d58d97961, 85a5f2cc79b74d5b9aefec2c215e466d; no voters:
I20260924 11:40:43.143163 2025 raft_consensus.cc:2781] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:43.143253 2025 raft_consensus.cc:482] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:43.143404 2025 raft_consensus.cc:3037] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:43.145962 2025 raft_consensus.cc:504] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.146555 2025 leader_election.cc:288] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 election: Requested vote from peers 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655), 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035)
I20260924 11:40:43.146816 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15fa46813d24fd0ae65157192350321" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a5f2cc79b74d5b9aefec2c215e466d"
I20260924 11:40:43.147058 1815 raft_consensus.cc:3037] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:43.146999 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15fa46813d24fd0ae65157192350321" candidate_uuid: "205b9030a5af45dbaee64c7d58d97961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d21b78c5fc45eb8b88bcfa13463110"
I20260924 11:40:43.147214 1954 raft_consensus.cc:3037] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:43.148917 1815 raft_consensus.cc:2454] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 1.
I20260924 11:40:43.149333 1602 leader_election.cc:302] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205b9030a5af45dbaee64c7d58d97961, 85a5f2cc79b74d5b9aefec2c215e466d; no voters:
I20260924 11:40:43.149545 2025 raft_consensus.cc:2781] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:43.149722 2025 raft_consensus.cc:686] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [term 1 LEADER]: Becoming Leader. State: Replica: 205b9030a5af45dbaee64c7d58d97961, State: Running, Role: LEADER
I20260924 11:40:43.149955 2025 consensus_queue.cc:237] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } }
I20260924 11:40:43.150302 1954 raft_consensus.cc:2454] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205b9030a5af45dbaee64c7d58d97961 in term 1.
I20260924 11:40:43.151893 2040 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: TxnStatusTablet state changed. Reason: New leader 205b9030a5af45dbaee64c7d58d97961. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.152251 2040 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:40:43.152940 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:40:43.153175 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:40:43.153275 1515 catalog_manager.cc:5495] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 205b9030a5af45dbaee64c7d58d97961 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } health_report { overall_health: UNKNOWN } } }
I20260924 11:40:43.170596 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.164892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.0.97.1:35765)
0924 11:40:43.165078 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.165082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.165159 (+ 77us) server_negotiation.cc:407] Connection header received
0924 11:40:43.165404 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.165407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.165465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.165537 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:43.165919 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.167131 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.168993 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.169398 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.169481 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.169993 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:43.170101 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:43.170162 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.170325 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.170383 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:40:43.170735 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.164795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35765 (local address 127.0.0.1:58432)
0924 11:40:43.165045 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:43.165082 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:43.165241 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:43.165592 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:43.165597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:43.165610 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:43.165787 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:43.165798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.167280 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.167285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:43.168861 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:43.168874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.169532 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.169538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:43.169776 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.169843 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:43.170193 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:43.170217 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 11:40:43.170304 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:40:43.170446 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:40:43.173817 1815 raft_consensus.cc:1261] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Refusing update from remote peer 205b9030a5af45dbaee64c7d58d97961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:43.173935 1954 raft_consensus.cc:1261] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Refusing update from remote peer 205b9030a5af45dbaee64c7d58d97961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:43.174501 2040 consensus_queue.cc:1035] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:43.174827 2025 consensus_queue.cc:1035] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:43.177861 2026 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: TxnStatusTablet state changed. Reason: New leader 205b9030a5af45dbaee64c7d58d97961. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.178154 2026 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.179430 2040 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.179689 2040 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:40:43.181124 2034 mvcc.cc:205] Tried to move back new op lower bound from 7332864176834539520 to 7332864176744546304. Current Snapshot: MvccSnapshot[applied={T|T < 7332864176834539520}]
I20260924 11:40:43.181610 2026 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.181800 2026 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 85a5f2cc79b74d5b9aefec2c215e466d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.186414 2024 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: TxnStatusTablet state changed. Reason: New leader 205b9030a5af45dbaee64c7d58d97961. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.186690 2024 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.188069 2029 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.188376 2029 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 205b9030a5af45dbaee64c7d58d97961: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:40:43.189953 2024 tablet_replica.cc:439] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "205b9030a5af45dbaee64c7d58d97961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205b9030a5af45dbaee64c7d58d97961" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35765 } } peers { permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 } } peers { permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 } } }
I20260924 11:40:43.190157 2024 tablet_replica.cc:442] T a15fa46813d24fd0ae65157192350321 P 54d21b78c5fc45eb8b88bcfa13463110: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:43.205322 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.198416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42893 (local address 127.0.97.1:35765)
0924 11:40:43.198619 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.198630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.199326 (+ 696us) server_negotiation.cc:407] Connection header received
0924 11:40:43.199610 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.199617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.199705 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.199801 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:43.200342 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.201384 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.202945 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.203691 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.203839 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.204194 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:43.204332 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:43.204423 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.204565 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.204639 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:40:43.205327 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.198448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35765 (local address 127.0.97.1:42893)
0924 11:40:43.199204 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:43.199221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:43.199362 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:43.199823 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:43.199826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:43.199834 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:43.200208 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:43.200217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.201497 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.201500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:43.202805 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:43.202816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.203863 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.203867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:43.203993 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.204039 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:43.204447 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:43.204455 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:40:43.204526 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 11:40:43.204617 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":405,"threads_started":1}
I20260924 11:40:43.213330 1954 raft_consensus.cc:1261] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Refusing update from remote peer 205b9030a5af45dbaee64c7d58d97961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:43.213888 1815 raft_consensus.cc:1261] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Refusing update from remote peer 205b9030a5af45dbaee64c7d58d97961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:43.213891 2040 consensus_queue.cc:1035] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d21b78c5fc45eb8b88bcfa13463110" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:43.214411 2040 consensus_queue.cc:1035] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a5f2cc79b74d5b9aefec2c215e466d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:43.223860 2033 mvcc.cc:205] Tried to move back new op lower bound from 7332864176997597184 to 7332864176018866176. Current Snapshot: MvccSnapshot[applied={T|T < 7332864176997597184}]
I20260924 11:40:43.270643 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:43.270867 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:43.273852 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f8024d2791a47f086ae4f716179cd10: Sending DeleteTablet for 3 replicas of tablet dc50fe5ea53d416e89d1ca28d377dbd9
I20260924 11:40:43.274479 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet dc50fe5ea53d416e89d1ca28d377dbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58416
I20260924 11:40:43.274575 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet dc50fe5ea53d416e89d1ca28d377dbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44924
I20260924 11:40:43.274726 2044 tablet_replica.cc:330] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: stopping tablet replica
I20260924 11:40:43.274679 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet dc50fe5ea53d416e89d1ca28d377dbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53696
I20260924 11:40:43.274927 2044 raft_consensus.cc:2229] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:40:43.274964 2042 tablet_replica.cc:330] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: stopping tablet replica
I20260924 11:40:43.275117 2042 raft_consensus.cc:2229] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:43.275209 2044 raft_consensus.cc:2258] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:43.275285 2042 raft_consensus.cc:2258] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:43.275470 2043 tablet_replica.cc:330] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: stopping tablet replica
I20260924 11:40:43.275653 2043 raft_consensus.cc:2229] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:43.275811 2043 raft_consensus.cc:2258] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:43.276362 2042 ts_tablet_manager.cc:1905] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:43.277245 2043 ts_tablet_manager.cc:1905] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:43.282577 2044 ts_tablet_manager.cc:1905] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:43.283784 2042 ts_tablet_manager.cc:1918] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:43.283962 2042 log.cc:1196] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/dc50fe5ea53d416e89d1ca28d377dbd9
I20260924 11:40:43.284410 2042 ts_tablet_manager.cc:1939] T dc50fe5ea53d416e89d1ca28d377dbd9 P 54d21b78c5fc45eb8b88bcfa13463110: Deleting consensus metadata
I20260924 11:40:43.285219 2043 ts_tablet_manager.cc:1918] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:43.285420 2043 log.cc:1196] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/dc50fe5ea53d416e89d1ca28d377dbd9
I20260924 11:40:43.285597 1501 catalog_manager.cc:4841] TS 54d21b78c5fc45eb8b88bcfa13463110 (127.0.97.3:39035): tablet dc50fe5ea53d416e89d1ca28d377dbd9 (table test-table [id=8ee49c7100a94d78bc45d9987ebc599a]) successfully deleted
I20260924 11:40:43.285986 2043 ts_tablet_manager.cc:1939] T dc50fe5ea53d416e89d1ca28d377dbd9 P 85a5f2cc79b74d5b9aefec2c215e466d: Deleting consensus metadata
I20260924 11:40:43.286995 1499 catalog_manager.cc:4841] TS 85a5f2cc79b74d5b9aefec2c215e466d (127.0.97.2:39655): tablet dc50fe5ea53d416e89d1ca28d377dbd9 (table test-table [id=8ee49c7100a94d78bc45d9987ebc599a]) successfully deleted
I20260924 11:40:43.290249 2044 ts_tablet_manager.cc:1918] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:43.290422 2044 log.cc:1196] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/dc50fe5ea53d416e89d1ca28d377dbd9
I20260924 11:40:43.290913 2044 ts_tablet_manager.cc:1939] T dc50fe5ea53d416e89d1ca28d377dbd9 P 205b9030a5af45dbaee64c7d58d97961: Deleting consensus metadata
I20260924 11:40:43.291937 1499 catalog_manager.cc:4841] TS 205b9030a5af45dbaee64c7d58d97961 (127.0.97.1:35765): tablet dc50fe5ea53d416e89d1ca28d377dbd9 (table test-table [id=8ee49c7100a94d78bc45d9987ebc599a]) successfully deleted
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:43.302779 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.287315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.0.97.1:35765)
0924 11:40:43.287580 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.287589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.288101 (+ 512us) server_negotiation.cc:407] Connection header received
0924 11:40:43.288318 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.288323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.288390 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.288503 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:43.289804 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.290775 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.292184 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.292649 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.292806 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.296878 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:43.296893 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:43.296908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:43.296934 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:43.299948 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.300702 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.300709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.300711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.300814 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.301283 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.301286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.301288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.301612 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:43.301838 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.302182 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.302383 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 11:40:43.303534 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58448
I20260924 11:40:43.328176 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.306713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.0.97.62:45229)
0924 11:40:43.306946 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.306954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.313069 (+ 6115us) server_negotiation.cc:407] Connection header received
0924 11:40:43.313233 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.313239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.313325 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.313436 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:43.314985 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.316041 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.317841 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.318540 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.318671 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.321705 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:43.321712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:43.321718 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:43.321757 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:43.325259 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.326006 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.326012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.326016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.326166 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.326579 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.326586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.326590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.326941 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:43.327203 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.327550 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.327766 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 11:40:43.328704 1515 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43444
I20260924 11:40:43.332813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1586
I20260924 11:40:43.350775 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.333830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.0.97.1:35765)
0924 11:40:43.334029 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.334036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.336609 (+ 2573us) server_negotiation.cc:407] Connection header received
0924 11:40:43.336763 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.336768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.336841 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.336933 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:43.338397 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.339564 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.341773 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.342297 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.342385 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.345093 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:43.345097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:43.345100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:43.345125 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:43.348289 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.348952 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.348957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.348960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.349080 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.349432 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.349437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.349440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.349788 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:43.350021 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.350303 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.350507 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 11:40:43.351245 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58462
W20260924 11:40:43.351706 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:40:43.351868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1586
I20260924 11:40:43.369078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1725
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:40:43.382427 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.369727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.0.97.2:39655)
0924 11:40:43.370230 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.370234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.370263 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:40:43.370312 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.370315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.370381 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.370496 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:43.371426 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.372241 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.373357 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.373765 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.373860 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.377507 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:43.377518 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:43.377530 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:43.377565 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:43.380294 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.380833 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.380839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.380841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.380979 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.381268 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.381270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.381272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.381562 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:43.381803 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.381995 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.382164 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
W20260924 11:40:43.383189 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44938
W20260924 11:40:43.383767 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:40:43.383963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1725
I20260924 11:40:43.401314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1864
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:40:43.414713 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:43.401917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.0.97.3:39035)
0924 11:40:43.402381 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:43.402385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:43.402398 (+ 13us) server_negotiation.cc:407] Connection header received
0924 11:40:43.402441 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:43.402444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:43.402500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:43.402609 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:43.403700 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.404435 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.405614 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:43.405983 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:43.406111 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:43.409749 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:43.409759 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:43.409771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:43.409796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:43.412285 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.412876 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.412887 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.412895 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.413015 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:43.413379 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:43.413388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:43.413390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:43.413658 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:43.413957 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:43.414193 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 11:40:43.414392 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
W20260924 11:40:43.415277 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706
W20260924 11:40:43.415757 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:40:43.415917 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1864
I20260924 11:40:43.431372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 1483
I20260924 11:40:43.432325 1555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:43.537434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 1483
2026-09-24T11:40:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4278 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[2080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[2080](info): set up 2 sockets
Sep 24 11:40:43 dist-test-slave-qhrh krb5kdc[2080](info): commencing operation
krb5kdc: starting...
W20260924 11:40:44.568212 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:40:44 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:40:44.731781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35753
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:45.028416 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:45.028648 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:45.028817 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:45.035668 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:45.035763 2093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:40:45.035849 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:45.035945 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:45.036033 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:45.036134 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:45.045117 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35753
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:45.046249 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:45.047461 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:45.058431 2099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:45.058480 2101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:45.058516 2098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:45.059453 2093 server_base.cc:1034] running on GCE node
I20260924 11:40:45.060107 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:45.061306 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:45.062528 2093 hybrid_clock.cc:648] HybridClock initialized: now 1790250045062437 us; error 74 us; skew 500 ppm
I20260924 11:40:45.062817 2093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:45 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:45.065747 2093 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:45.067052 2093 webserver.cc:460] Webserver started at http://127.0.97.62:37843/ using document root <none> and password file <none>
I20260924 11:40:45.067677 2093 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:45.067795 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:45.068095 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:45.070734 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "11130bf92230466aacede0df37c45663"
format_stamp: "Formatted at 2026-09-24 11:40:45 on dist-test-slave-qhrh"
server_key: "179e8289757925d463a6cd59036d7124"
server_key_iv: "2364e00c5cf6e5a856fedc996f603cad"
server_key_version: "encryptionkey@0"
I20260924 11:40:45.071435 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "11130bf92230466aacede0df37c45663"
format_stamp: "Formatted at 2026-09-24 11:40:45 on dist-test-slave-qhrh"
server_key: "179e8289757925d463a6cd59036d7124"
server_key_iv: "2364e00c5cf6e5a856fedc996f603cad"
server_key_version: "encryptionkey@0"
I20260924 11:40:45.076452 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:40:45.079444 2108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:45.080679 2093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:40:45.080835 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "11130bf92230466aacede0df37c45663"
format_stamp: "Formatted at 2026-09-24 11:40:45 on dist-test-slave-qhrh"
server_key: "179e8289757925d463a6cd59036d7124"
server_key_iv: "2364e00c5cf6e5a856fedc996f603cad"
server_key_version: "encryptionkey@0"
I20260924 11:40:45.081012 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:45.097601 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:45.101197 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:45.101435 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:45.120962 2093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35753
I20260924 11:40:45.120996 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35753 every 8 connection(s)
I20260924 11:40:45.122004 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:45.125200 2170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:45.125223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 2093
I20260924 11:40:45.125435 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:45.125988 388 external_mini_cluster.cc:1428] Setting key 3db4a8a35f530ffe498ce77329475b0e
I20260924 11:40:45.135871 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Bootstrap starting.
Sep 24 11:40:45 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:45.138546 2170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:45.139406 2170 log.cc:824] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:45.141644 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: No bootstrap required, opened a new log
I20260924 11:40:45.145602 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:45.127768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.0.97.62:35753)
0924 11:40:45.128494 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:45.128504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:45.128545 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:40:45.130644 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:45.130661 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:45.130985 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:45.131336 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:45.132626 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.133866 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.135051 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.135652 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.135825 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:45.139382 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:45.139388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:45.139402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:45.139434 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:45.142124 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:45.142745 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:45.142751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:45.142756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:45.142930 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:45.143294 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:45.143301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:45.143304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:45.143767 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:45.144108 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:45.144458 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 11:40:45.144789 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":377,"threads_started":1}
I20260924 11:40:45.148767 2170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130bf92230466aacede0df37c45663" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35753 } }
I20260924 11:40:45.149063 2170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:45.149156 2170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11130bf92230466aacede0df37c45663, State: Initialized, Role: FOLLOWER
I20260924 11:40:45.149595 2170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130bf92230466aacede0df37c45663" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35753 } }
I20260924 11:40:45.149821 2170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:45.149958 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:45.150120 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:45.152302 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130bf92230466aacede0df37c45663" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35753 } }
I20260924 11:40:45.152735 2170 leader_election.cc:302] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11130bf92230466aacede0df37c45663; no voters:
I20260924 11:40:45.153256 2170 leader_election.cc:288] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:45.153347 2175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:45.154093 2175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [term 1 LEADER]: Becoming Leader. State: Replica: 11130bf92230466aacede0df37c45663, State: Running, Role: LEADER
I20260924 11:40:45.154572 2175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130bf92230466aacede0df37c45663" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35753 } }
I20260924 11:40:45.155107 2170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:45.157075 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11130bf92230466aacede0df37c45663. Latest consensus state: current_term: 1 leader_uuid: "11130bf92230466aacede0df37c45663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130bf92230466aacede0df37c45663" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35753 } } }
I20260924 11:40:45.157570 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:45.159358 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11130bf92230466aacede0df37c45663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130bf92230466aacede0df37c45663" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35753 } } }
I20260924 11:40:45.159574 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:45.159915 2183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:45.164733 2183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:45.173825 2183 catalog_manager.cc:1347] Generated new cluster ID: 128148bc61c74fa980ab69d0d8cda438
I20260924 11:40:45.173935 2183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:45.182107 2183 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:45.183228 2183 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:45.196060 2183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Generated new TSK 0
I20260924 11:40:45.196760 2183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:40:45.240819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35753
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:45.562194 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:45.562435 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:45.562590 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:45.569438 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:45.569527 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:45.569717 2196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:40:45.569794 2196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:40:45.569921 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:40:45.578998 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:45.580106 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:45.581321 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:45.593729 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:45.593804 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:45.593725 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:45.594379 2196 server_base.cc:1034] running on GCE node
I20260924 11:40:45.594928 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:45.595746 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:45.597050 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790250045596990 us; error 45 us; skew 500 ppm
I20260924 11:40:45.597364 2196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:45 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:45.600395 2196 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:40:45.601655 2196 webserver.cc:460] Webserver started at http://127.0.97.1:45915/ using document root <none> and password file <none>
I20260924 11:40:45.602191 2196 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:45.602279 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:45.602497 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:45.605259 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "c25f75af54ca42198eb2aa89142a677e"
format_stamp: "Formatted at 2026-09-24 11:40:45 on dist-test-slave-qhrh"
server_key: "e6d5708a0f4d13d5c4327b0f7571f223"
server_key_iv: "f0fbb35de6050fd381c73a5d336834d5"
server_key_version: "encryptionkey@0"
I20260924 11:40:45.605928 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c25f75af54ca42198eb2aa89142a677e"
format_stamp: "Formatted at 2026-09-24 11:40:45 on dist-test-slave-qhrh"
server_key: "e6d5708a0f4d13d5c4327b0f7571f223"
server_key_iv: "f0fbb35de6050fd381c73a5d336834d5"
server_key_version: "encryptionkey@0"
I20260924 11:40:45.610450 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:40:45.613487 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:45.614675 2196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:40:45.614790 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "c25f75af54ca42198eb2aa89142a677e"
format_stamp: "Formatted at 2026-09-24 11:40:45 on dist-test-slave-qhrh"
server_key: "e6d5708a0f4d13d5c4327b0f7571f223"
server_key_iv: "f0fbb35de6050fd381c73a5d336834d5"
server_key_version: "encryptionkey@0"
I20260924 11:40:45.614939 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:45.638216 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:45.642176 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:45.642439 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:45.644969 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:45.645120 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:45.645231 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:45.645313 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:40:45 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:45.666137 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:45.645276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46707 (local address 127.0.97.62:35753)
0924 11:40:45.645460 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:45.645465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:45.646871 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 11:40:45.649748 (+ 2877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:45.649757 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:45.649850 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:45.649969 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:45.652331 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.653245 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.655133 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.655879 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.656021 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:45.659925 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:45.659931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:45.659935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:45.660013 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:45.663127 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:45.664030 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:45.664035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:45.664039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:45.664220 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:45.664644 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:45.664648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:45.664651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:45.664877 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:45.665007 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:45.665614 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 11:40:45.665809 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:40:45.667022 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:45.645822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.97.1:46707)
0924 11:40:45.646670 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:45.646729 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:45.649428 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:45.650238 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:45.650253 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:45.651122 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:45.652103 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:45.652133 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.653464 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.653479 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:45.655004 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:45.655014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.656187 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.656196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:45.656362 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:45.657000 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:45.657020 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:45.659725 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:45.663259 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:45.663267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:45.663282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:45.663901 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:45.664331 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:45.664337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:45.664341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:45.664535 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:45.664996 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:45.665002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:45.665293 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 11:40:45.665527 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 11:40:45.665978 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":313,"threads_started":1}
I20260924 11:40:45.693100 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:45.679481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49135 (local address 127.0.97.62:35753)
0924 11:40:45.679645 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:45.679652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:45.680572 (+ 920us) server_negotiation.cc:407] Connection header received
0924 11:40:45.680834 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:45.680844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:45.680894 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:45.680941 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:45.682374 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.683377 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.685122 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.685557 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.685685 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:45.687299 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:45.687303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:45.687306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:45.687332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:45.690678 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:45.691296 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:45.691301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:45.691304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:45.691438 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:45.691758 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:45.691764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:45.691768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:45.692082 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:45.692260 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:45.692731 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 11:40:45.692866 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 11:40:45.693722 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:45.679517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.97.1:49135)
0924 11:40:45.680450 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:45.680492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:45.680674 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:45.681157 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:45.681163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:45.681702 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:45.682241 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:45.682251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.683507 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.683513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:45.684971 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:45.684981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:45.685728 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:45.685733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:45.685889 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:45.686476 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:45.686492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:45.687123 (+ 631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:45.690794 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:45.690797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:45.690800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:45.691169 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:45.691539 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:45.691542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:45.691544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:45.691649 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:45.692222 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:45.692229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:45.692479 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 11:40:45.692672 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 11:40:45.693080 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":525,"threads_started":1}
I20260924 11:40:45.706014 2196 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42719
I20260924 11:40:45.706079 2330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42719 every 8 connection(s)
I20260924 11:40:45.707118 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:40:45.708906 2331 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35753
I20260924 11:40:45.709105 2331 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:45.709667 2331 heartbeater.cc:511] Master 127.0.97.62:35753 requested a full tablet report, sending...
I20260924 11:40:45.711359 2125 ts_manager.cc:194] Registered new tserver with Master: c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719)
I20260924 11:40:45.712718 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46707
I20260924 11:40:45.717020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 2196
I20260924 11:40:45.717271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:40:45.717872 388 external_mini_cluster.cc:1428] Setting key ccff5aa0256739ffee1851255f5bd809
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:40:45.754873 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35753
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:46.046342 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:46.046618 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:46.046787 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:46.053148 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:46.053277 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:46.053429 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:40:46.053521 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:40:46.053639 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:40:46.062681 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:46.063771 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:46.064931 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:46.077216 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:46.077296 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:46.077253 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:46.078330 2335 server_base.cc:1034] running on GCE node
I20260924 11:40:46.078837 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:46.079640 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:46.080814 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790250046080790 us; error 54 us; skew 500 ppm
I20260924 11:40:46.081097 2335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:46.083892 2335 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:40:46.085176 2335 webserver.cc:460] Webserver started at http://127.0.97.2:36985/ using document root <none> and password file <none>
I20260924 11:40:46.085810 2335 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:46.085906 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:46.086138 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:46.088686 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "99e479f763114cfb9616d6f783e4e380"
format_stamp: "Formatted at 2026-09-24 11:40:46 on dist-test-slave-qhrh"
server_key: "55b6a782067d6b2d120a1932063896c7"
server_key_iv: "2a25bd106f19cacfd7652ab5f2e9da82"
server_key_version: "encryptionkey@0"
I20260924 11:40:46.089351 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "99e479f763114cfb9616d6f783e4e380"
format_stamp: "Formatted at 2026-09-24 11:40:46 on dist-test-slave-qhrh"
server_key: "55b6a782067d6b2d120a1932063896c7"
server_key_iv: "2a25bd106f19cacfd7652ab5f2e9da82"
server_key_version: "encryptionkey@0"
I20260924 11:40:46.093761 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:40:46.096809 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:46.097911 2335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:40:46.098033 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "99e479f763114cfb9616d6f783e4e380"
format_stamp: "Formatted at 2026-09-24 11:40:46 on dist-test-slave-qhrh"
server_key: "55b6a782067d6b2d120a1932063896c7"
server_key_iv: "2a25bd106f19cacfd7652ab5f2e9da82"
server_key_version: "encryptionkey@0"
I20260924 11:40:46.098173 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:46.113461 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:46.116762 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:46.117007 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:46.119424 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:46.119509 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:46.119653 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:46.119742 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:46.144073 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.119901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59705 (local address 127.0.97.62:35753)
0924 11:40:46.120125 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.120131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.121336 (+ 1205us) server_negotiation.cc:407] Connection header received
0924 11:40:46.123841 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.123850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.123957 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.124103 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:46.125819 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.127362 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.130689 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.131614 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.131770 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.136113 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:46.136123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:46.136130 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:46.136201 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:46.140453 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.141266 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.141274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.141280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.141424 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.141969 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.141976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.141981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.142323 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:46.142464 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.143395 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.143654 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:40:46.145502 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.120456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.97.2:59705)
0924 11:40:46.121173 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.121212 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.123530 (+ 2318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.124353 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.124362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.124969 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:46.125621 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.125643 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.127537 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.127543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.129723 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.129736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.131846 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.131852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.132100 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.132821 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:46.132846 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:46.135820 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:46.140579 (+ 4759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.140584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.140592 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.141106 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.141568 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.141581 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.141587 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.141826 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.142460 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:46.142469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:46.142904 (+ 435us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.143274 (+ 370us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.143848 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":305,"threads_started":1}
I20260924 11:40:46.169760 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.152653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54201 (local address 127.0.97.62:35753)
0924 11:40:46.152856 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.152864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.153707 (+ 843us) server_negotiation.cc:407] Connection header received
0924 11:40:46.153903 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.153910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.154001 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.154131 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:46.155662 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.156832 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.159311 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.160049 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.160191 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.162904 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:46.162908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:46.162911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:46.162941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:46.166309 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.167289 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.167297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.167302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.167454 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.167985 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.168005 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.168010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.168455 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:46.168603 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.169236 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.169487 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:40:46.170136 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.152703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.97.2:54201)
0924 11:40:46.153591 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.153616 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.153770 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.154179 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.154190 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.154850 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:46.155468 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.155484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.157019 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.157026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.159115 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.159137 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.160680 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.160690 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.160907 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.161540 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:46.161574 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:46.162746 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:46.166506 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.166514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.166519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.167152 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.167622 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.167629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.167636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.167817 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.168606 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:46.168613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:46.168880 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.169122 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.169486 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":533,"threads_started":1}
I20260924 11:40:46.172003 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37889
I20260924 11:40:46.172068 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37889 every 8 connection(s)
I20260924 11:40:46.173153 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:40:46.175341 2470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35753
I20260924 11:40:46.175581 2470 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:46.176129 2470 heartbeater.cc:511] Master 127.0.97.62:35753 requested a full tablet report, sending...
I20260924 11:40:46.177282 2125 ts_manager.cc:194] Registered new tserver with Master: 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889)
I20260924 11:40:46.178110 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59705
I20260924 11:40:46.178202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 2335
I20260924 11:40:46.178371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:40:46.178855 388 external_mini_cluster.cc:1428] Setting key 7f9c8da82c574107382033182c12bced
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:40:46.215853 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35753
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:46.521427 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:46.521667 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:46.521755 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:46.528461 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:46.528561 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:46.528645 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:40:46.528700 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:40:46.528760 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:40:46.537627 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:46.538704 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:46.539888 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:46.551925 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:46.551999 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:46.551977 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:46.556233 2474 server_base.cc:1034] running on GCE node
I20260924 11:40:46.556821 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:46.557619 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:46.558804 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790250046558763 us; error 37 us; skew 500 ppm
I20260924 11:40:46.559104 2474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:46.572837 2474 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:40:46.574249 2474 webserver.cc:460] Webserver started at http://127.0.97.3:40695/ using document root <none> and password file <none>
I20260924 11:40:46.574877 2474 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:46.574997 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:46.575289 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:46.577971 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "dad4548aad6f473985d19a5134fe0378"
format_stamp: "Formatted at 2026-09-24 11:40:46 on dist-test-slave-qhrh"
server_key: "0ab706e820d7b1d011d844732baff27e"
server_key_iv: "33a18669d90a7f380248fbfa988b78ce"
server_key_version: "encryptionkey@0"
I20260924 11:40:46.578711 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dad4548aad6f473985d19a5134fe0378"
format_stamp: "Formatted at 2026-09-24 11:40:46 on dist-test-slave-qhrh"
server_key: "0ab706e820d7b1d011d844732baff27e"
server_key_iv: "33a18669d90a7f380248fbfa988b78ce"
server_key_version: "encryptionkey@0"
I20260924 11:40:46.583205 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:40:46.586381 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:46.587558 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:46.587716 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "dad4548aad6f473985d19a5134fe0378"
format_stamp: "Formatted at 2026-09-24 11:40:46 on dist-test-slave-qhrh"
server_key: "0ab706e820d7b1d011d844732baff27e"
server_key_iv: "33a18669d90a7f380248fbfa988b78ce"
server_key_version: "encryptionkey@0"
I20260924 11:40:46.587895 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:46.619022 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:46.622822 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:46.623104 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:46.625533 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:46.625664 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:46.625782 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:46.625865 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:46.647955 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.625847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50339 (local address 127.0.97.62:35753)
0924 11:40:46.626023 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.626029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.627366 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 11:40:46.629526 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.629532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.629621 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.629746 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:46.631651 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.632873 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.635263 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.635669 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.635796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.640589 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:46.640599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:46.640607 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:46.640656 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:46.644600 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.645680 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.645688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.645694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.645853 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.646248 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.646255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.646261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.646647 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:46.646796 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.647414 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.647648 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 11:40:46.648815 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.626384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.97.3:50339)
0924 11:40:46.627178 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.627221 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.629259 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.630006 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.630017 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.630746 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:46.631436 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.631453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.633106 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.633112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.635088 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.635100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.635891 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.635897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.636118 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.636796 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:46.636815 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:46.640263 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:46.644761 (+ 4498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.644772 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.644786 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.645516 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.645944 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.645947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.645949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.646119 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.646769 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:46.646776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:46.647095 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.647342 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.647770 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":362,"threads_started":1}
I20260924 11:40:46.666172 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.653698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45347 (local address 127.0.97.62:35753)
0924 11:40:46.653835 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.653839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.654630 (+ 791us) server_negotiation.cc:407] Connection header received
0924 11:40:46.654863 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.654868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.654976 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.655102 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:46.655959 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.656848 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.657960 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.658332 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.658453 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.660472 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:46.660477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:46.660481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:46.660515 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:46.663533 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.664430 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.664437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.664440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.664606 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.665024 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.665027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.665029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.665246 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:46.665353 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.665830 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.665955 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:40:46.666775 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.653828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.97.3:45347)
0924 11:40:46.654509 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.654524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.654713 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.655118 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.655121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.655565 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:46.655835 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.655842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.656957 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.656963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.657828 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.657838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.658489 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.658495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.658648 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.659282 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:46.659297 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:46.660289 (+ 992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:46.663680 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.663683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.663686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.664283 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.664728 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.664735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.664741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.664907 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.665355 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:46.665362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:46.665586 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.665761 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.666151 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":264,"threads_started":1}
I20260924 11:40:46.679535 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42625
I20260924 11:40:46.679615 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42625 every 8 connection(s)
I20260924 11:40:46.680665 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:40:46.682337 2609 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35753
I20260924 11:40:46.682572 2609 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:46.683133 2609 heartbeater.cc:511] Master 127.0.97.62:35753 requested a full tablet report, sending...
I20260924 11:40:46.684386 2125 ts_manager.cc:194] Registered new tserver with Master: dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:46.685156 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50339
I20260924 11:40:46.690090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 2474
I20260924 11:40:46.690268 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:40:46.690696 388 external_mini_cluster.cc:1428] Setting key 209d2cc20afd9bfa3bf26e590185d854
I20260924 11:40:46.693221 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:40:46.714938 2331 heartbeater.cc:503] Master 127.0.97.62:35753 was elected leader, sending a full tablet report...
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:46.736035 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.724430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.0.97.62:35753)
0924 11:40:46.724584 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.724588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.725022 (+ 434us) server_negotiation.cc:407] Connection header received
0924 11:40:46.725155 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.725158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.725230 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.725306 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:46.726484 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.727133 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.728261 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.728631 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.728716 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.731952 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:46.731956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:46.731958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:46.731981 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:46.734195 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.734704 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.734708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.734710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.734801 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.735080 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.735082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.735084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.735307 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:46.735473 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.735702 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.735839 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:40:46.739322 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:40:46.745595 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:46.777428 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.756040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37889 (local address 127.0.0.1:33674)
0924 11:40:46.766297 (+ 10257us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.766337 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.766489 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.769312 (+ 2823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.769320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.769356 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:46.769840 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.769854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.772178 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.772184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.774706 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.774723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.776452 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.776457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.776639 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.776683 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.776811 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.776974 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10026,"mutex_wait_us":1,"thread_start_us":14319,"threads_started":1}
I20260924 11:40:46.778017 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.766713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.0.1:46568)
0924 11:40:46.770194 (+ 3481us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.770211 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.770305 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.771481 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.771489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.771525 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:46.771938 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.771949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.773737 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.773742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.775510 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.775521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.777243 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.777267 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.777477 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.777512 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.777604 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.777718 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3363,"thread_start_us":2426,"threads_started":1}
I20260924 11:40:46.778103 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.756226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.97.2:37889)
0924 11:40:46.768726 (+ 12500us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.768736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.768765 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:40:46.768867 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.768874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.769100 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.770056 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:46.770236 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.772055 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.774872 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.776246 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.776352 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.776518 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.777676 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.777834 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12355,"thread_start_us":12242,"threads_started":1}
I20260924 11:40:46.778713 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.766979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.0.97.1:42719)
0924 11:40:46.770970 (+ 3991us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.770978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.771002 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:40:46.771094 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.771100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.771280 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.771458 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:46.772094 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.773555 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.775663 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.776810 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.776906 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.777107 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.778309 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.778470 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3838,"thread_start_us":3709,"threads_started":1}
I20260924 11:40:46.779446 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.756335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42625 (local address 127.0.0.1:38498)
0924 11:40:46.768950 (+ 12615us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.768965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.769054 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.772516 (+ 3462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.772521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.772542 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:46.772925 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.772933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.774752 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.774759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.776809 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.776824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.778983 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.778988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.779091 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.779111 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.779155 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.779221 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12523,"thread_start_us":9597,"threads_started":1}
I20260924 11:40:46.780128 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.757071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.0.97.3:42625)
0924 11:40:46.772027 (+ 14956us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.772034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.772073 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:40:46.772201 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.772207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.772361 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.772527 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:46.773145 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.774628 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.776970 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.778764 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.779654 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.779791 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.779837 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.779923 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14815,"thread_start_us":14665,"threads_started":1}
I20260924 11:40:46.781251 2401 tablet_service.cc:1467] Processing CreateTablet for tablet c4f15606734e4d449de671bdb0562e32 (DEFAULT_TABLE table=test-table [id=6aa4160ec2664695bf3f6395697c76ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:46.782570 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f15606734e4d449de671bdb0562e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:46.782541 2261 tablet_service.cc:1467] Processing CreateTablet for tablet c4f15606734e4d449de671bdb0562e32 (DEFAULT_TABLE table=test-table [id=6aa4160ec2664695bf3f6395697c76ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:46.783116 2539 tablet_service.cc:1467] Processing CreateTablet for tablet c4f15606734e4d449de671bdb0562e32 (DEFAULT_TABLE table=test-table [id=6aa4160ec2664695bf3f6395697c76ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:46.783619 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f15606734e4d449de671bdb0562e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:46.784408 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f15606734e4d449de671bdb0562e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:46.796321 2629 tablet_bootstrap.cc:492] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Bootstrap starting.
I20260924 11:40:46.797966 2630 tablet_bootstrap.cc:492] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Bootstrap starting.
I20260924 11:40:46.798646 2628 tablet_bootstrap.cc:492] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Bootstrap starting.
I20260924 11:40:46.800947 2630 tablet_bootstrap.cc:654] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:46.801487 2628 tablet_bootstrap.cc:654] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:46.802002 2630 log.cc:824] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:46.802033 2629 tablet_bootstrap.cc:654] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:46.802662 2628 log.cc:824] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:46.803133 2629 log.cc:824] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:46.804814 2630 tablet_bootstrap.cc:492] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: No bootstrap required, opened a new log
I20260924 11:40:46.805053 2630 ts_tablet_manager.cc:1397] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260924 11:40:46.805567 2629 tablet_bootstrap.cc:492] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: No bootstrap required, opened a new log
I20260924 11:40:46.805790 2629 ts_tablet_manager.cc:1397] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 11:40:46.806879 2628 tablet_bootstrap.cc:492] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: No bootstrap required, opened a new log
I20260924 11:40:46.807107 2628 ts_tablet_manager.cc:1397] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 11:40:46.812099 2629 raft_consensus.cc:348] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.812495 2629 raft_consensus.cc:374] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:46.812628 2629 raft_consensus.cc:729] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Initialized, Role: FOLLOWER
I20260924 11:40:46.813177 2629 consensus_queue.cc:260] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.814107 2470 heartbeater.cc:503] Master 127.0.97.62:35753 was elected leader, sending a full tablet report...
I20260924 11:40:46.814668 2629 ts_tablet_manager.cc:1428] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260924 11:40:46.814961 2630 raft_consensus.cc:348] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.815316 2630 raft_consensus.cc:374] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:46.815400 2630 raft_consensus.cc:729] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad4548aad6f473985d19a5134fe0378, State: Initialized, Role: FOLLOWER
I20260924 11:40:46.815909 2630 consensus_queue.cc:260] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.817447 2630 ts_tablet_manager.cc:1428] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260924 11:40:46.817643 2609 heartbeater.cc:503] Master 127.0.97.62:35753 was elected leader, sending a full tablet report...
I20260924 11:40:46.817957 2628 raft_consensus.cc:348] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.818329 2628 raft_consensus.cc:374] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:46.818420 2628 raft_consensus.cc:729] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Initialized, Role: FOLLOWER
I20260924 11:40:46.818920 2628 consensus_queue.cc:260] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.820420 2628 ts_tablet_manager.cc:1428] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260924 11:40:46.882076 2634 raft_consensus.cc:482] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:46.882325 2634 raft_consensus.cc:504] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.883796 2634 leader_election.cc:288] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625), c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719)
I20260924 11:40:46.892565 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.884080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42625 (local address 127.0.97.2:38489)
0924 11:40:46.884693 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.884706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.884898 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.885390 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.885395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.885423 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:46.885856 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.885871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.887245 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.887249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.889691 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.889703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.891235 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.891241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.891519 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.891620 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.891760 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.891951 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":117,"spinlock_wait_cycles":18688,"thread_start_us":335,"threads_started":1}
I20260924 11:40:46.892627 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.884020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38489 (local address 127.0.97.3:42625)
0924 11:40:46.884269 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.884273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.884811 (+ 538us) server_negotiation.cc:407] Connection header received
0924 11:40:46.885052 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.885059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.885151 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.885213 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:46.885990 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.887106 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.889831 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.890937 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.891073 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.891170 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.891899 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.892408 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:40:46.892835 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.884912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56833 (local address 127.0.97.1:42719)
0924 11:40:46.885118 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.885124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.885453 (+ 329us) server_negotiation.cc:407] Connection header received
0924 11:40:46.885613 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.885617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.885687 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.885751 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:46.886299 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.887303 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.889834 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.890960 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.891073 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.891181 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.891830 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.892068 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:40:46.893162 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.884816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.97.2:56833)
0924 11:40:46.885388 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.885402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.885497 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.885789 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.885795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.885824 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:46.886152 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.886158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.887438 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.887444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.889690 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.889703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.891228 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.891235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.891504 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.891604 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.891760 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.891951 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":170,"thread_start_us":279,"threads_started":1}
I20260924 11:40:46.894260 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f15606734e4d449de671bdb0562e32" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378" is_pre_election: true
I20260924 11:40:46.894837 2564 raft_consensus.cc:2454] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 0.
I20260924 11:40:46.895570 2351 leader_election.cc:302] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99e479f763114cfb9616d6f783e4e380, dad4548aad6f473985d19a5134fe0378; no voters:
I20260924 11:40:46.895479 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f15606734e4d449de671bdb0562e32" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25f75af54ca42198eb2aa89142a677e" is_pre_election: true
I20260924 11:40:46.895963 2634 raft_consensus.cc:2781] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:46.895968 2286 raft_consensus.cc:2454] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 0.
I20260924 11:40:46.896128 2634 raft_consensus.cc:482] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:46.896266 2634 raft_consensus.cc:3037] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:46.898516 2634 raft_consensus.cc:504] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.899091 2634 leader_election.cc:288] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 election: Requested vote from peers dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625), c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719)
I20260924 11:40:46.899461 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f15606734e4d449de671bdb0562e32" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378"
I20260924 11:40:46.899547 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f15606734e4d449de671bdb0562e32" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25f75af54ca42198eb2aa89142a677e"
I20260924 11:40:46.899657 2564 raft_consensus.cc:3037] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:46.899714 2286 raft_consensus.cc:3037] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:46.901837 2286 raft_consensus.cc:2454] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 1.
I20260924 11:40:46.902230 2351 leader_election.cc:302] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99e479f763114cfb9616d6f783e4e380, c25f75af54ca42198eb2aa89142a677e; no voters:
I20260924 11:40:46.902237 2564 raft_consensus.cc:2454] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 1.
I20260924 11:40:46.902495 2634 raft_consensus.cc:2781] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:46.902935 2634 raft_consensus.cc:686] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 1 LEADER]: Becoming Leader. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Running, Role: LEADER
I20260924 11:40:46.903370 2634 consensus_queue.cc:237] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } }
I20260924 11:40:46.907838 2124 catalog_manager.cc:5495] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e479f763114cfb9616d6f783e4e380 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "99e479f763114cfb9616d6f783e4e380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20260924 11:40:46.925377 2471 tablet.cc:2369] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:40:46.931942 2610 tablet.cc:2369] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 11:40:46 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
W20260924 11:40:46.959137 2332 tablet.cc:2369] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:40:46.961858 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.950078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35753 (local address 127.0.0.1:37294)
0924 11:40:46.950571 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:46.950585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:46.950718 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:46.951000 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:46.951003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:46.951726 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:46.951952 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.951960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.952888 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.952891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.953850 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:46.953859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.954498 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.954502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:46.954617 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.955312 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:46.955327 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:46.957331 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:46.959873 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.959880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.959885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.960352 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.960654 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:46.960656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:46.960659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:46.960770 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:46.961170 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:46.961175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:46.961294 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 11:40:46.961409 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 11:40:46.961581 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":298,"threads_started":1}
I20260924 11:40:46.961953 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:46.950000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.0.97.62:35753)
0924 11:40:46.950162 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:46.950166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:46.950678 (+ 512us) server_negotiation.cc:407] Connection header received
0924 11:40:46.950815 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:46.950818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:46.950879 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:46.950945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:46.952065 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.952785 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.953957 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:46.954361 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:46.954442 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:46.957503 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:46.957507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:46.957510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:46.957535 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:46.959764 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.960455 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.960459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.960461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.960565 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:46.960850 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:46.960854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:46.960855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:46.961082 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:46.961175 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:46.961485 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 11:40:46.961672 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":70}
I20260924 11:40:46.966256 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37294:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:40:46.967265 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:46.973469 2261 tablet_service.cc:1467] Processing CreateTablet for tablet a0c0468354e7486ea73b5bbb5016fd72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f6f832663c34c78951dc61472f82919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:40:46.974273 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c0468354e7486ea73b5bbb5016fd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:46.973799 2401 tablet_service.cc:1467] Processing CreateTablet for tablet a0c0468354e7486ea73b5bbb5016fd72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f6f832663c34c78951dc61472f82919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:40:46.973798 2539 tablet_service.cc:1467] Processing CreateTablet for tablet a0c0468354e7486ea73b5bbb5016fd72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f6f832663c34c78951dc61472f82919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:40:46.974606 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c0468354e7486ea73b5bbb5016fd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:46.974606 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c0468354e7486ea73b5bbb5016fd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:46.979490 2628 tablet_bootstrap.cc:492] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: Bootstrap starting.
I20260924 11:40:46.981112 2628 tablet_bootstrap.cc:654] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:46.981384 2629 tablet_bootstrap.cc:492] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: Bootstrap starting.
I20260924 11:40:46.981556 2630 tablet_bootstrap.cc:492] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: Bootstrap starting.
I20260924 11:40:46.982935 2629 tablet_bootstrap.cc:654] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:46.983032 2630 tablet_bootstrap.cc:654] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:46.983247 2628 tablet_bootstrap.cc:492] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: No bootstrap required, opened a new log
I20260924 11:40:46.983374 2628 ts_tablet_manager.cc:1397] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 11:40:46.984337 2628 raft_consensus.cc:348] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:46.984678 2628 raft_consensus.cc:374] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:46.984786 2628 raft_consensus.cc:729] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Initialized, Role: FOLLOWER
I20260924 11:40:46.985085 2628 consensus_queue.cc:260] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:46.985497 2630 tablet_bootstrap.cc:492] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: No bootstrap required, opened a new log
I20260924 11:40:46.985520 2629 tablet_bootstrap.cc:492] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: No bootstrap required, opened a new log
I20260924 11:40:46.985622 2630 ts_tablet_manager.cc:1397] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 11:40:46.985641 2629 ts_tablet_manager.cc:1397] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 11:40:46.985633 2628 ts_tablet_manager.cc:1428] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:40:46.985476 2636 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:46.985839 2636 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:46.986334 2629 raft_consensus.cc:348] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:46.986332 2630 raft_consensus.cc:348] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:46.986575 2629 raft_consensus.cc:374] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:46.986574 2630 raft_consensus.cc:374] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:46.986663 2630 raft_consensus.cc:729] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad4548aad6f473985d19a5134fe0378, State: Initialized, Role: FOLLOWER
I20260924 11:40:46.986663 2629 raft_consensus.cc:729] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Initialized, Role: FOLLOWER
I20260924 11:40:46.986857 2630 consensus_queue.cc:260] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:46.986857 2629 consensus_queue.cc:260] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:46.987373 2630 ts_tablet_manager.cc:1428] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 11:40:46.987435 2629 ts_tablet_manager.cc:1428] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:40:46.987285 2634 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:46.987600 2634 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:46.987941 2635 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:46.988303 2635 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:47.026507 2636 raft_consensus.cc:482] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:47.026674 2636 raft_consensus.cc:504] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.027954 2636 leader_election.cc:288] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889), dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:47.036814 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.028207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52305 (local address 127.0.97.2:37889)
0924 11:40:47.028401 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.028407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.029398 (+ 991us) server_negotiation.cc:407] Connection header received
0924 11:40:47.029525 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.029530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.029604 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.029694 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:47.030419 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.031450 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.034492 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.035687 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.035820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.035913 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.036392 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.036516 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:40:47.037072 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.028335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42625 (local address 127.0.97.1:56439)
0924 11:40:47.029229 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:47.029248 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:47.029392 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:47.029724 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:47.029728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:47.029754 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:47.030179 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.030195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.031591 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.031594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.034331 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.034344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.035954 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.035961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.036222 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.036273 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 11:40:47.036361 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:40:47.036554 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":398,"thread_start_us":411,"threads_started":1}
I20260924 11:40:47.037103 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.028390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37889 (local address 127.0.97.1:52305)
0924 11:40:47.029312 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:47.029325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:47.029411 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:47.029724 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:47.029728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:47.029876 (+ 148us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:47.030281 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.030293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.031588 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.031593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.034330 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.034344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.035971 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.035975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.036217 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.036271 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 11:40:47.036361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 11:40:47.036551 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":216,"spinlock_wait_cycles":22912,"thread_start_us":451,"threads_started":1}
I20260924 11:40:47.037417 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.028299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56439 (local address 127.0.97.3:42625)
0924 11:40:47.028459 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.028466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.029331 (+ 865us) server_negotiation.cc:407] Connection header received
0924 11:40:47.029534 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.029538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.029620 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.029701 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:47.030369 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.031482 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.034493 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.035705 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.035820 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.035905 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.037026 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.037172 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:40:47.037909 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c0468354e7486ea73b5bbb5016fd72" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378" is_pre_election: true
I20260924 11:40:47.038130 2564 raft_consensus.cc:2454] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 0.
I20260924 11:40:47.038147 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c0468354e7486ea73b5bbb5016fd72" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e479f763114cfb9616d6f783e4e380" is_pre_election: true
I20260924 11:40:47.038520 2425 raft_consensus.cc:2454] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 0.
I20260924 11:40:47.038532 2212 leader_election.cc:302] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25f75af54ca42198eb2aa89142a677e, dad4548aad6f473985d19a5134fe0378; no voters:
I20260924 11:40:47.038898 2636 raft_consensus.cc:2781] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:47.039011 2636 raft_consensus.cc:482] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:47.039125 2636 raft_consensus.cc:3037] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:47.041142 2636 raft_consensus.cc:504] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.041620 2636 leader_election.cc:288] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 election: Requested vote from peers 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889), dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:47.042006 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c0468354e7486ea73b5bbb5016fd72" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e479f763114cfb9616d6f783e4e380"
I20260924 11:40:47.042001 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c0468354e7486ea73b5bbb5016fd72" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378"
I20260924 11:40:47.042174 2425 raft_consensus.cc:3037] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:47.042199 2564 raft_consensus.cc:3037] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:47.044975 2564 raft_consensus.cc:2454] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 1.
I20260924 11:40:47.044987 2425 raft_consensus.cc:2454] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 1.
I20260924 11:40:47.045346 2212 leader_election.cc:302] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25f75af54ca42198eb2aa89142a677e, dad4548aad6f473985d19a5134fe0378; no voters:
I20260924 11:40:47.045588 2636 raft_consensus.cc:2781] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:47.045953 2636 raft_consensus.cc:686] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [term 1 LEADER]: Becoming Leader. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Running, Role: LEADER
I20260924 11:40:47.046388 2636 consensus_queue.cc:237] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.049299 2652 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: TxnStatusTablet state changed. Reason: New leader c25f75af54ca42198eb2aa89142a677e. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.049669 2652 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:40:47.050477 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:40:47.050731 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:40:47.051184 2124 catalog_manager.cc:5495] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e reported cstate change: term changed from 0 to 1, leader changed from <none> to c25f75af54ca42198eb2aa89142a677e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } health_report { overall_health: UNKNOWN } } }
I20260924 11:40:47.085278 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.079295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.97.1:42719)
0924 11:40:47.079452 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.079459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.079779 (+ 320us) server_negotiation.cc:407] Connection header received
0924 11:40:47.080009 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.080013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.080110 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.080278 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:47.080639 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.081314 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.083080 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.083487 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.083581 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.084074 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.084726 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.084818 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.084987 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.085065 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:40:47.085332 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.079347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.0.1:46574)
0924 11:40:47.079695 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:47.079715 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:47.079850 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:47.080320 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:47.080325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:47.080336 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:47.080504 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.080513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.081425 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.081430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.082949 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.082964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.083625 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.083631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.083851 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.083914 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.084823 (+ 909us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.084841 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 11:40:47.084921 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 11:40:47.085062 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260924 11:40:47.092002 2425 raft_consensus.cc:1261] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Refusing update from remote peer c25f75af54ca42198eb2aa89142a677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:47.092155 2564 raft_consensus.cc:1261] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Refusing update from remote peer c25f75af54ca42198eb2aa89142a677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:47.092940 2636 consensus_queue.cc:1035] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:47.093300 2652 consensus_queue.cc:1035] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:47.107120 2635 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: TxnStatusTablet state changed. Reason: New leader c25f75af54ca42198eb2aa89142a677e. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.108296 2635 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:47.116654 2636 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.116935 2636 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:40:47.119783 2634 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: TxnStatusTablet state changed. Reason: New leader c25f75af54ca42198eb2aa89142a677e. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.120097 2634 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:47.121701 2635 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.121917 2635 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P dad4548aad6f473985d19a5134fe0378: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:47.122262 2658 mvcc.cc:205] Tried to move back new op lower bound from 7332864192879755264 to 7332864192705470464. Current Snapshot: MvccSnapshot[applied={T|T < 7332864192879755264}]
I20260924 11:40:47.132690 2652 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.132932 2652 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P c25f75af54ca42198eb2aa89142a677e: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:40:47.141037 2634 tablet_replica.cc:439] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } } }
I20260924 11:40:47.141285 2634 tablet_replica.cc:442] T a0c0468354e7486ea73b5bbb5016fd72 P 99e479f763114cfb9616d6f783e4e380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:40:47.151970 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.145312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.97.2:37889)
0924 11:40:47.145449 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.145454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.145511 (+ 57us) server_negotiation.cc:407] Connection header received
0924 11:40:47.145712 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.145717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.145798 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.145938 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:47.146488 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.147591 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.149379 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.149936 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.150034 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.150581 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.151104 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.151206 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.151632 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.151702 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:40:47.166992 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.161735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44333 (local address 127.0.97.1:42719)
0924 11:40:47.161876 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.161879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.162390 (+ 511us) server_negotiation.cc:407] Connection header received
0924 11:40:47.162605 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.162609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.162682 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.162771 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:47.163256 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.163980 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.165315 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.165776 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.165917 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.166317 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.166404 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.166498 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.166639 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.166709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:40:47.167091 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.161746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.97.2:44333)
0924 11:40:47.162295 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:47.162312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:47.162468 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:47.162794 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:47.162810 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:47.162821 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:47.163123 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.163130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.164110 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.164116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.165164 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.165191 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.165946 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.165952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.166077 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.166120 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.166499 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.166511 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 11:40:47.166619 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:40:47.166735 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":272,"threads_started":1}
I20260924 11:40:47.179852 2564 raft_consensus.cc:1261] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Refusing update from remote peer 99e479f763114cfb9616d6f783e4e380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:47.180006 2286 raft_consensus.cc:1261] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Refusing update from remote peer 99e479f763114cfb9616d6f783e4e380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:47.180523 2639 consensus_queue.cc:1035] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:47.180907 2634 consensus_queue.cc:1035] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:47.194139 2659 mvcc.cc:205] Tried to move back new op lower bound from 7332864193243430912 to 7332864192120258560. Current Snapshot: MvccSnapshot[applied={T|T < 7332864193243430912}]
I20260924 11:40:47.215996 2657 mvcc.cc:205] Tried to move back new op lower bound from 7332864193243430912 to 7332864192120258560. Current Snapshot: MvccSnapshot[applied={T|T < 7332864193243430912}]
I20260924 11:40:47.226614 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.219324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55051 (local address 127.0.97.2:37889)
0924 11:40:47.219477 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.219483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.221414 (+ 1931us) server_negotiation.cc:407] Connection header received
0924 11:40:47.221556 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.221558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.221617 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.221681 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:47.222306 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.223054 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.225033 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.225422 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.225553 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.225943 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.226024 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.226117 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.226264 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.226349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 11:40:47.226778 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.218980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37889 (local address 127.0.97.1:55051)
0924 11:40:47.221328 (+ 2348us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:47.221345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:47.221466 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:47.221718 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:47.221722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:47.221734 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:47.222071 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.222081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.223209 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.223214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.224882 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.224893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.225594 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.225601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.225739 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.225785 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.226122 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.226134 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 11:40:47.226224 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 11:40:47.226376 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2189,"thread_start_us":1816,"threads_started":1}
I20260924 11:40:47.259814 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:47.260056 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:47.263381 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Sending DeleteTablet for 3 replicas of tablet c4f15606734e4d449de671bdb0562e32
I20260924 11:40:47.264515 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f15606734e4d449de671bdb0562e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260924 11:40:47.265112 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f15606734e4d449de671bdb0562e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260924 11:40:47.265367 2677 tablet_replica.cc:330] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: stopping tablet replica
I20260924 11:40:47.265772 2677 raft_consensus.cc:2229] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:40:47.266034 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f15606734e4d449de671bdb0562e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46568
I20260924 11:40:47.266623 2677 raft_consensus.cc:2258] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:47.268713 2677 ts_tablet_manager.cc:1905] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:47.272434 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:40:47.272994 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:47.276779 2677 ts_tablet_manager.cc:1918] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:47.276993 2677 log.cc:1196] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/c4f15606734e4d449de671bdb0562e32
I20260924 11:40:47.277624 2677 ts_tablet_manager.cc:1939] T c4f15606734e4d449de671bdb0562e32 P 99e479f763114cfb9616d6f783e4e380: Deleting consensus metadata
I20260924 11:40:47.279505 2110 catalog_manager.cc:4841] TS 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889): tablet c4f15606734e4d449de671bdb0562e32 (table test-table [id=6aa4160ec2664695bf3f6395697c76ee]) successfully deleted
I20260924 11:40:47.280314 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf29b3b60714fd1af52e0ad4d66045b (DEFAULT_TABLE table=test-table [id=a9d429e452f24aafb4313bc040c00cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:47.280452 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf29b3b60714fd1af52e0ad4d66045b (DEFAULT_TABLE table=test-table [id=a9d429e452f24aafb4313bc040c00cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:47.280687 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf29b3b60714fd1af52e0ad4d66045b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:47.280855 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf29b3b60714fd1af52e0ad4d66045b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:47.287246 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf29b3b60714fd1af52e0ad4d66045b (DEFAULT_TABLE table=test-table [id=a9d429e452f24aafb4313bc040c00cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:47.287719 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf29b3b60714fd1af52e0ad4d66045b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:47.288048 2629 tablet_bootstrap.cc:492] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Bootstrap starting.
I20260924 11:40:47.289817 2629 tablet_bootstrap.cc:654] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:47.293658 2628 tablet_bootstrap.cc:492] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Bootstrap starting.
I20260924 11:40:47.295214 2628 tablet_bootstrap.cc:654] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:47.297538 2630 tablet_bootstrap.cc:492] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Bootstrap starting.
I20260924 11:40:47.299103 2630 tablet_bootstrap.cc:654] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:47.300413 2679 tablet_replica.cc:330] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: stopping tablet replica
I20260924 11:40:47.302245 2629 tablet_bootstrap.cc:492] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: No bootstrap required, opened a new log
I20260924 11:40:47.302399 2629 ts_tablet_manager.cc:1397] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260924 11:40:47.302994 2629 raft_consensus.cc:348] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.303221 2629 raft_consensus.cc:374] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:47.303306 2629 raft_consensus.cc:729] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Initialized, Role: FOLLOWER
I20260924 11:40:47.303567 2629 consensus_queue.cc:260] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.304522 2678 tablet_replica.cc:330] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: stopping tablet replica
I20260924 11:40:47.304906 2678 raft_consensus.cc:2229] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:47.305249 2678 raft_consensus.cc:2258] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:47.305701 2679 raft_consensus.cc:2229] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:47.306016 2679 raft_consensus.cc:2258] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:47.304116 2629 ts_tablet_manager.cc:1428] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:40:47.307799 2630 tablet_bootstrap.cc:492] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: No bootstrap required, opened a new log
I20260924 11:40:47.307968 2630 ts_tablet_manager.cc:1397] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260924 11:40:47.308009 2679 ts_tablet_manager.cc:1905] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:47.308719 2630 raft_consensus.cc:348] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.308992 2630 raft_consensus.cc:374] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:47.309109 2630 raft_consensus.cc:729] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad4548aad6f473985d19a5134fe0378, State: Initialized, Role: FOLLOWER
I20260924 11:40:47.309355 2630 consensus_queue.cc:260] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.310344 2630 ts_tablet_manager.cc:1428] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:40:47.311369 2678 ts_tablet_manager.cc:1905] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:47.314634 2679 ts_tablet_manager.cc:1918] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:47.314782 2679 log.cc:1196] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/c4f15606734e4d449de671bdb0562e32
I20260924 11:40:47.315260 2679 ts_tablet_manager.cc:1939] T c4f15606734e4d449de671bdb0562e32 P c25f75af54ca42198eb2aa89142a677e: Deleting consensus metadata
I20260924 11:40:47.316643 2109 catalog_manager.cc:4841] TS c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719): tablet c4f15606734e4d449de671bdb0562e32 (table test-table [id=6aa4160ec2664695bf3f6395697c76ee]) successfully deleted
I20260924 11:40:47.317776 2628 tablet_bootstrap.cc:492] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: No bootstrap required, opened a new log
I20260924 11:40:47.317930 2628 ts_tablet_manager.cc:1397] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.008s
I20260924 11:40:47.318457 2628 raft_consensus.cc:348] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.318667 2628 raft_consensus.cc:374] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:47.318750 2628 raft_consensus.cc:729] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Initialized, Role: FOLLOWER
I20260924 11:40:47.318996 2628 consensus_queue.cc:260] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.319563 2628 ts_tablet_manager.cc:1428] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 11:40:47.324491 2678 ts_tablet_manager.cc:1918] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:47.324661 2678 log.cc:1196] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/c4f15606734e4d449de671bdb0562e32
I20260924 11:40:47.325083 2678 ts_tablet_manager.cc:1939] T c4f15606734e4d449de671bdb0562e32 P dad4548aad6f473985d19a5134fe0378: Deleting consensus metadata
I20260924 11:40:47.326313 2109 catalog_manager.cc:4841] TS dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625): tablet c4f15606734e4d449de671bdb0562e32 (table test-table [id=6aa4160ec2664695bf3f6395697c76ee]) successfully deleted
I20260924 11:40:47.590127 2652 raft_consensus.cc:482] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:47.590342 2652 raft_consensus.cc:504] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.590909 2652 leader_election.cc:288] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889), dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:47.591434 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf29b3b60714fd1af52e0ad4d66045b" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e479f763114cfb9616d6f783e4e380" is_pre_election: true
I20260924 11:40:47.591456 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf29b3b60714fd1af52e0ad4d66045b" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378" is_pre_election: true
I20260924 11:40:47.591713 2564 raft_consensus.cc:2454] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 0.
I20260924 11:40:47.591711 2425 raft_consensus.cc:2454] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 0.
I20260924 11:40:47.592190 2212 leader_election.cc:302] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25f75af54ca42198eb2aa89142a677e, dad4548aad6f473985d19a5134fe0378; no voters:
I20260924 11:40:47.592516 2652 raft_consensus.cc:2781] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:47.592620 2652 raft_consensus.cc:482] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:47.592730 2652 raft_consensus.cc:3037] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:47.595669 2652 raft_consensus.cc:504] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.596292 2652 leader_election.cc:288] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 election: Requested vote from peers 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889), dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:47.596738 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf29b3b60714fd1af52e0ad4d66045b" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e479f763114cfb9616d6f783e4e380"
I20260924 11:40:47.596750 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf29b3b60714fd1af52e0ad4d66045b" candidate_uuid: "c25f75af54ca42198eb2aa89142a677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378"
I20260924 11:40:47.596918 2564 raft_consensus.cc:3037] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:47.596917 2425 raft_consensus.cc:3037] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:47.598910 2425 raft_consensus.cc:2454] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 1.
I20260924 11:40:47.599290 2213 leader_election.cc:302] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99e479f763114cfb9616d6f783e4e380, c25f75af54ca42198eb2aa89142a677e; no voters:
I20260924 11:40:47.599529 2652 raft_consensus.cc:2781] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:47.599709 2652 raft_consensus.cc:686] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 1 LEADER]: Becoming Leader. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Running, Role: LEADER
I20260924 11:40:47.599951 2652 consensus_queue.cc:237] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:47.600219 2564 raft_consensus.cc:2454] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25f75af54ca42198eb2aa89142a677e in term 1.
I20260924 11:40:47.602460 2123 catalog_manager.cc:5495] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e reported cstate change: term changed from 0 to 1, leader changed from <none> to c25f75af54ca42198eb2aa89142a677e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c25f75af54ca42198eb2aa89142a677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } health_report { overall_health: UNKNOWN } } }
I20260924 11:40:47.662917 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.657998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.97.1:42719)
0924 11:40:47.658204 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.658213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.658503 (+ 290us) server_negotiation.cc:407] Connection header received
0924 11:40:47.658686 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.658690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.658755 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.658839 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:47.659235 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.659915 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.661354 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.661766 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.661899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.662237 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.662333 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.662425 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.662576 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.662643 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:40:47.675514 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.670516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46295 (local address 127.0.97.1:42719)
0924 11:40:47.670650 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:47.670653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:47.670900 (+ 247us) server_negotiation.cc:407] Connection header received
0924 11:40:47.671088 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:47.671091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:47.671140 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:47.671203 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:47.671556 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.672263 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.674027 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.674391 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.674526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.674872 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.674954 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.675040 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:47.675169 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 11:40:47.675224 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":39}
I20260924 11:40:47.675514 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:47.670545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.97.1:46295)
0924 11:40:47.670818 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:47.670837 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:47.670954 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:47.671251 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:47.671256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:47.671268 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:47.671433 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.671442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.672383 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.672388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.673905 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:47.673919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:47.674536 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:47.674541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:47.674708 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:47.674736 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:47.675040 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:47.675052 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 11:40:47.675143 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 11:40:47.675260 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 11:40:47.688442 2564 raft_consensus.cc:1261] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Refusing update from remote peer c25f75af54ca42198eb2aa89142a677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:47.689044 2425 raft_consensus.cc:1261] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Refusing update from remote peer c25f75af54ca42198eb2aa89142a677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:47.689527 2652 consensus_queue.cc:1035] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:47.689857 2652 consensus_queue.cc:1035] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:40:47.724879 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 11:40:49.317178 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:49.317365 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:49.321167 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Sending DeleteTablet for 3 replicas of tablet 0cf29b3b60714fd1af52e0ad4d66045b
I20260924 11:40:49.321818 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf29b3b60714fd1af52e0ad4d66045b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46568
I20260924 11:40:49.322043 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf29b3b60714fd1af52e0ad4d66045b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260924 11:40:49.322077 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf29b3b60714fd1af52e0ad4d66045b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260924 11:40:49.336591 2714 tablet_replica.cc:330] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: stopping tablet replica
I20260924 11:40:49.336854 2714 raft_consensus.cc:2229] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:49.337117 2714 raft_consensus.cc:2258] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:49.338312 2714 ts_tablet_manager.cc:1905] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:49.344930 2713 tablet_replica.cc:330] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: stopping tablet replica
I20260924 11:40:49.345170 2713 raft_consensus.cc:2229] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:49.345399 2713 raft_consensus.cc:2258] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:49.345916 2712 tablet_replica.cc:330] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: stopping tablet replica
I20260924 11:40:49.346098 2712 raft_consensus.cc:2229] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:40:49.346293 2714 ts_tablet_manager.cc:1918] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:40:49.346338 2713 ts_tablet_manager.cc:1905] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:49.346451 2714 log.cc:1196] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/0cf29b3b60714fd1af52e0ad4d66045b
I20260924 11:40:49.346671 2712 raft_consensus.cc:2258] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:49.346890 2714 ts_tablet_manager.cc:1939] T 0cf29b3b60714fd1af52e0ad4d66045b P dad4548aad6f473985d19a5134fe0378: Deleting consensus metadata
I20260924 11:40:49.347699 2712 ts_tablet_manager.cc:1905] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:49.347852 2109 catalog_manager.cc:4841] TS dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625): tablet 0cf29b3b60714fd1af52e0ad4d66045b (table test-table [id=a9d429e452f24aafb4313bc040c00cc9]) successfully deleted
I20260924 11:40:49.352501 2713 ts_tablet_manager.cc:1918] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:40:49.352620 2713 log.cc:1196] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/0cf29b3b60714fd1af52e0ad4d66045b
I20260924 11:40:49.352663 2712 ts_tablet_manager.cc:1918] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:40:49.352797 2712 log.cc:1196] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/0cf29b3b60714fd1af52e0ad4d66045b
I20260924 11:40:49.353014 2713 ts_tablet_manager.cc:1939] T 0cf29b3b60714fd1af52e0ad4d66045b P 99e479f763114cfb9616d6f783e4e380: Deleting consensus metadata
I20260924 11:40:49.353252 2712 ts_tablet_manager.cc:1939] T 0cf29b3b60714fd1af52e0ad4d66045b P c25f75af54ca42198eb2aa89142a677e: Deleting consensus metadata
I20260924 11:40:49.353892 2110 catalog_manager.cc:4841] TS 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889): tablet 0cf29b3b60714fd1af52e0ad4d66045b (table test-table [id=a9d429e452f24aafb4313bc040c00cc9]) successfully deleted
I20260924 11:40:49.354270 2109 catalog_manager.cc:4841] TS c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719): tablet 0cf29b3b60714fd1af52e0ad4d66045b (table test-table [id=a9d429e452f24aafb4313bc040c00cc9]) successfully deleted
Sep 24 11:40:49 dist-test-slave-qhrh krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:40:49 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:49.379709 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.367757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.97.62:35753)
0924 11:40:49.368324 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:49.368328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:49.368346 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:40:49.368451 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:49.368454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:49.368509 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:49.368591 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:49.369852 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.370615 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.371739 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.372060 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.372191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.375478 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:49.375483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:49.375486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:49.375512 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:49.377814 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:49.378280 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:49.378283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:49.378285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:49.378378 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:49.378731 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:49.378734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:49.378736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:49.378970 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:49.379123 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:49.379343 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 11:40:49.379478 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":280,"threads_started":1}
I20260924 11:40:49.382757 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:40:49.383754 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:49.389551 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf (DEFAULT_TABLE table=test-table [id=992d16480a6b4feeb8a95008fd6448da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:49.389808 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf (DEFAULT_TABLE table=test-table [id=992d16480a6b4feeb8a95008fd6448da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:49.389930 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:49.389926 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf (DEFAULT_TABLE table=test-table [id=992d16480a6b4feeb8a95008fd6448da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:49.390156 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:49.390221 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:49.394667 2723 tablet_bootstrap.cc:492] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Bootstrap starting.
I20260924 11:40:49.395887 2723 tablet_bootstrap.cc:654] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:49.396795 2725 tablet_bootstrap.cc:492] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Bootstrap starting.
I20260924 11:40:49.397598 2724 tablet_bootstrap.cc:492] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Bootstrap starting.
I20260924 11:40:49.398193 2725 tablet_bootstrap.cc:654] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:49.398343 2723 tablet_bootstrap.cc:492] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: No bootstrap required, opened a new log
I20260924 11:40:49.398466 2723 ts_tablet_manager.cc:1397] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 11:40:49.398988 2724 tablet_bootstrap.cc:654] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:49.399159 2723 raft_consensus.cc:348] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.399394 2723 raft_consensus.cc:374] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:49.399479 2723 raft_consensus.cc:729] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad4548aad6f473985d19a5134fe0378, State: Initialized, Role: FOLLOWER
I20260924 11:40:49.400107 2725 tablet_bootstrap.cc:492] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: No bootstrap required, opened a new log
I20260924 11:40:49.399953 2723 consensus_queue.cc:260] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.400404 2725 ts_tablet_manager.cc:1397] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:40:49.401144 2725 raft_consensus.cc:348] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.401345 2725 raft_consensus.cc:374] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:49.401444 2725 raft_consensus.cc:729] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Initialized, Role: FOLLOWER
I20260924 11:40:49.401677 2725 consensus_queue.cc:260] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.402221 2725 ts_tablet_manager.cc:1428] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:40:49.402987 2723 ts_tablet_manager.cc:1428] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260924 11:40:49.405771 2724 tablet_bootstrap.cc:492] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: No bootstrap required, opened a new log
I20260924 11:40:49.405897 2724 ts_tablet_manager.cc:1397] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260924 11:40:49.406672 2724 raft_consensus.cc:348] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.406903 2724 raft_consensus.cc:374] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:49.406963 2724 raft_consensus.cc:729] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Initialized, Role: FOLLOWER
I20260924 11:40:49.407125 2724 consensus_queue.cc:260] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.407773 2724 ts_tablet_manager.cc:1428] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 11:40:49.682621 2729 raft_consensus.cc:482] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:49.682834 2729 raft_consensus.cc:504] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.684782 2729 leader_election.cc:288] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719), 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889)
I20260924 11:40:49.703981 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.685025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.97.3:60903)
0924 11:40:49.696385 (+ 11360us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:49.696403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:49.696554 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:49.697018 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:49.697023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:49.697051 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:49.697399 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.697413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.699068 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.699074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.701659 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.701673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.703189 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.703195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.703392 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.703425 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:40:49.703524 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 11:40:49.703649 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11236,"thread_start_us":11664,"threads_started":1}
I20260924 11:40:49.704005 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.685229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60903 (local address 127.0.97.1:42719)
0924 11:40:49.692529 (+ 7300us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:49.692537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:49.696484 (+ 3947us) server_negotiation.cc:407] Connection header received
0924 11:40:49.696720 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:49.696728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:49.696831 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:49.696986 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:49.697621 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.698845 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.701833 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.703021 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.703148 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.703282 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:49.703555 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 11:40:49.703690 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7142,"thread_start_us":7011,"threads_started":1}
I20260924 11:40:49.704875 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776dc5d8ce2f47e1adf8bc9e577ec7bf" candidate_uuid: "dad4548aad6f473985d19a5134fe0378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25f75af54ca42198eb2aa89142a677e" is_pre_election: true
I20260924 11:40:49.705092 2286 raft_consensus.cc:2454] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dad4548aad6f473985d19a5134fe0378 in term 0.
I20260924 11:40:49.705578 2490 leader_election.cc:302] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25f75af54ca42198eb2aa89142a677e, dad4548aad6f473985d19a5134fe0378; no voters:
I20260924 11:40:49.705965 2729 raft_consensus.cc:2781] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:49.706079 2729 raft_consensus.cc:482] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:49.706183 2729 raft_consensus.cc:3037] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:49.706595 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.685174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37889 (local address 127.0.97.3:56983)
0924 11:40:49.699608 (+ 14434us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:49.699628 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:49.699778 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:49.700106 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:49.700110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:49.700131 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:40:49.700938 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.700949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.702315 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.702320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.704292 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.704303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.705680 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.705689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.705924 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.705977 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:40:49.706085 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:40:49.706218 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14303,"thread_start_us":14476,"threads_started":1}
I20260924 11:40:49.706929 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.685489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56983 (local address 127.0.97.2:37889)
0924 11:40:49.699090 (+ 13601us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:49.699117 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:49.699695 (+ 578us) server_negotiation.cc:407] Connection header received
0924 11:40:49.699905 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:49.699911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:49.699979 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:49.700083 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:40:49.701148 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.702173 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.704439 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.705480 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.705627 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.705745 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:49.706283 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 11:40:49.706443 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13480,"thread_start_us":13655,"threads_started":1}
I20260924 11:40:49.707399 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776dc5d8ce2f47e1adf8bc9e577ec7bf" candidate_uuid: "dad4548aad6f473985d19a5134fe0378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e479f763114cfb9616d6f783e4e380" is_pre_election: true
I20260924 11:40:49.707621 2424 raft_consensus.cc:2454] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dad4548aad6f473985d19a5134fe0378 in term 0.
I20260924 11:40:49.709177 2729 raft_consensus.cc:504] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.709754 2729 leader_election.cc:288] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [CANDIDATE]: Term 1 election: Requested vote from peers c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719), 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889)
I20260924 11:40:49.710141 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776dc5d8ce2f47e1adf8bc9e577ec7bf" candidate_uuid: "dad4548aad6f473985d19a5134fe0378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25f75af54ca42198eb2aa89142a677e"
I20260924 11:40:49.710321 2286 raft_consensus.cc:3037] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:49.710299 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776dc5d8ce2f47e1adf8bc9e577ec7bf" candidate_uuid: "dad4548aad6f473985d19a5134fe0378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e479f763114cfb9616d6f783e4e380"
I20260924 11:40:49.710472 2424 raft_consensus.cc:3037] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:49.713011 2424 raft_consensus.cc:2454] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dad4548aad6f473985d19a5134fe0378 in term 1.
I20260924 11:40:49.713311 2286 raft_consensus.cc:2454] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dad4548aad6f473985d19a5134fe0378 in term 1.
I20260924 11:40:49.713409 2491 leader_election.cc:302] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99e479f763114cfb9616d6f783e4e380, dad4548aad6f473985d19a5134fe0378; no voters:
I20260924 11:40:49.713677 2729 raft_consensus.cc:2781] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:49.714221 2729 raft_consensus.cc:686] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 1 LEADER]: Becoming Leader. State: Replica: dad4548aad6f473985d19a5134fe0378, State: Running, Role: LEADER
I20260924 11:40:49.714586 2729 consensus_queue.cc:237] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.718291 2124 catalog_manager.cc:5495] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 reported cstate change: term changed from 0 to 1, leader changed from <none> to dad4548aad6f473985d19a5134fe0378 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "dad4548aad6f473985d19a5134fe0378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } health_report { overall_health: HEALTHY } } }
I20260924 11:40:49.770288 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.764423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.0.97.3:42625)
0924 11:40:49.764939 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:49.764943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:49.764958 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:40:49.765021 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:49.765024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:49.765081 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:49.765243 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:49.765677 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.766444 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.767725 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.768125 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.768296 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.768743 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:49.769691 (+ 948us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:49.769846 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:49.769954 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 11:40:49.770021 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":279,"threads_started":1}
I20260924 11:40:49.782958 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.778430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40603 (local address 127.0.97.1:42719)
0924 11:40:49.778565 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:49.778569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:49.778988 (+ 419us) server_negotiation.cc:407] Connection header received
0924 11:40:49.779107 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:49.779111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:49.779165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:49.779234 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:49.779827 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.780558 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.781754 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.782083 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.782165 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.782496 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:49.782577 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:49.782645 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:49.782743 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 11:40:49.782791 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 11:40:49.783077 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.778298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42719 (local address 127.0.97.3:40603)
0924 11:40:49.778912 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:49.778927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:49.779020 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:49.779255 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:49.779258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:49.779266 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:49.779700 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.779710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.780687 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.780690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.781644 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.781654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.782211 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.782215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.782332 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.782373 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:49.782652 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:49.782659 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:40:49.782716 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 11:40:49.782797 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":336,"threads_started":1}
I20260924 11:40:49.789660 2424 raft_consensus.cc:1261] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Refusing update from remote peer dad4548aad6f473985d19a5134fe0378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:49.789673 2286 raft_consensus.cc:1261] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Refusing update from remote peer dad4548aad6f473985d19a5134fe0378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:49.790282 2735 consensus_queue.cc:1035] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:49.790550 2729 consensus_queue.cc:1035] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:49.813800 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.808672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54815 (local address 127.0.97.3:42625)
0924 11:40:49.808817 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:49.808822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:49.809182 (+ 360us) server_negotiation.cc:407] Connection header received
0924 11:40:49.809380 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:49.809384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:49.809450 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:49.809537 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:49.809904 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.810579 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.812018 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.812602 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.812747 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.813121 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:49.813224 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:49.813313 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:49.813471 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 11:40:49.813536 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:40:49.813906 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:49.808633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42625 (local address 127.0.97.1:54815)
0924 11:40:49.809095 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:49.809124 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:49.809249 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:49.809549 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:49.809554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:49.809564 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:40:49.809778 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.809786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.810727 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.810732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.811866 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:49.811877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:49.812782 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:49.812791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:49.812921 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:49.812952 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:40:49.813321 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:40:49.813333 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 11:40:49.813453 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 11:40:49.813574 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":250,"threads_started":1}
I20260924 11:40:49.838148 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:49.838354 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:49.841317 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Sending DeleteTablet for 3 replicas of tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf
I20260924 11:40:49.842005 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260924 11:40:49.842285 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260924 11:40:49.842523 2713 tablet_replica.cc:330] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: stopping tablet replica
I20260924 11:40:49.842713 2713 raft_consensus.cc:2229] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:49.842962 2714 tablet_replica.cc:330] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: stopping tablet replica
I20260924 11:40:49.843140 2714 raft_consensus.cc:2229] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:40:49.843104 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 11:40:49.843569 2714 raft_consensus.cc:2258] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 11:40:49.843756 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:40:49.843860 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46568
I20260924 11:40:49.844098 2712 tablet_replica.cc:330] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: stopping tablet replica
I20260924 11:40:49.844311 2712 raft_consensus.cc:2229] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:49.844476 2712 raft_consensus.cc:2258] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:49.844718 2713 raft_consensus.cc:2258] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:49.845556 2714 ts_tablet_manager.cc:1905] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:49.845721 2713 ts_tablet_manager.cc:1905] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:49.848286 2712 ts_tablet_manager.cc:1905] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:49.850044 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae648c1f2624981b524889a108e7c7e (DEFAULT_TABLE table=test-table [id=d6f5d365dcb24af9930d9907d500fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:49.850400 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae648c1f2624981b524889a108e7c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:49.855108 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae648c1f2624981b524889a108e7c7e (DEFAULT_TABLE table=test-table [id=d6f5d365dcb24af9930d9907d500fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:49.855405 2714 ts_tablet_manager.cc:1918] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:49.855504 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae648c1f2624981b524889a108e7c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:49.855561 2714 log.cc:1196] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/776dc5d8ce2f47e1adf8bc9e577ec7bf
I20260924 11:40:49.855932 2714 ts_tablet_manager.cc:1939] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P dad4548aad6f473985d19a5134fe0378: Deleting consensus metadata
I20260924 11:40:49.856456 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae648c1f2624981b524889a108e7c7e (DEFAULT_TABLE table=test-table [id=d6f5d365dcb24af9930d9907d500fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:40:49.856810 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae648c1f2624981b524889a108e7c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:49.856920 2109 catalog_manager.cc:4841] TS dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625): tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf (table test-table [id=992d16480a6b4feeb8a95008fd6448da]) successfully deleted
I20260924 11:40:49.858639 2712 ts_tablet_manager.cc:1918] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:49.858768 2712 log.cc:1196] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/776dc5d8ce2f47e1adf8bc9e577ec7bf
I20260924 11:40:49.859315 2712 ts_tablet_manager.cc:1939] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P c25f75af54ca42198eb2aa89142a677e: Deleting consensus metadata
I20260924 11:40:49.860289 2109 catalog_manager.cc:4841] TS c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719): tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf (table test-table [id=992d16480a6b4feeb8a95008fd6448da]) successfully deleted
I20260924 11:40:49.861850 2723 tablet_bootstrap.cc:492] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Bootstrap starting.
I20260924 11:40:49.863785 2724 tablet_bootstrap.cc:492] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Bootstrap starting.
I20260924 11:40:49.863891 2723 tablet_bootstrap.cc:654] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:49.864812 2713 ts_tablet_manager.cc:1918] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:40:49.864935 2713 log.cc:1196] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/776dc5d8ce2f47e1adf8bc9e577ec7bf
I20260924 11:40:49.865218 2725 tablet_bootstrap.cc:492] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Bootstrap starting.
I20260924 11:40:49.865341 2724 tablet_bootstrap.cc:654] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:49.865993 2713 ts_tablet_manager.cc:1939] T 776dc5d8ce2f47e1adf8bc9e577ec7bf P 99e479f763114cfb9616d6f783e4e380: Deleting consensus metadata
I20260924 11:40:49.866689 2725 tablet_bootstrap.cc:654] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:49.868284 2110 catalog_manager.cc:4841] TS 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889): tablet 776dc5d8ce2f47e1adf8bc9e577ec7bf (table test-table [id=992d16480a6b4feeb8a95008fd6448da]) successfully deleted
I20260924 11:40:49.869910 2724 tablet_bootstrap.cc:492] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: No bootstrap required, opened a new log
I20260924 11:40:49.870118 2724 ts_tablet_manager.cc:1397] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 11:40:49.870982 2724 raft_consensus.cc:348] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.872490 2724 raft_consensus.cc:374] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:49.872655 2724 raft_consensus.cc:729] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Initialized, Role: FOLLOWER
I20260924 11:40:49.873003 2724 consensus_queue.cc:260] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.873804 2724 ts_tablet_manager.cc:1428] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:40:49.875371 2723 tablet_bootstrap.cc:492] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: No bootstrap required, opened a new log
I20260924 11:40:49.875524 2723 ts_tablet_manager.cc:1397] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260924 11:40:49.876214 2723 raft_consensus.cc:348] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.876435 2723 raft_consensus.cc:374] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:49.876510 2723 raft_consensus.cc:729] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad4548aad6f473985d19a5134fe0378, State: Initialized, Role: FOLLOWER
I20260924 11:40:49.876776 2723 consensus_queue.cc:260] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.877310 2723 ts_tablet_manager.cc:1428] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:49.879627 2725 tablet_bootstrap.cc:492] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: No bootstrap required, opened a new log
I20260924 11:40:49.879773 2725 ts_tablet_manager.cc:1397] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260924 11:40:49.880486 2725 raft_consensus.cc:348] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.880708 2725 raft_consensus.cc:374] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:49.880787 2725 raft_consensus.cc:729] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25f75af54ca42198eb2aa89142a677e, State: Initialized, Role: FOLLOWER
I20260924 11:40:49.881083 2725 consensus_queue.cc:260] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:49.881567 2725 ts_tablet_manager.cc:1428] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:50.202142 2730 raft_consensus.cc:482] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:40:50.202358 2730 raft_consensus.cc:504] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:50.203033 2730 leader_election.cc:288] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719), dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:50.203574 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae648c1f2624981b524889a108e7c7e" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378" is_pre_election: true
I20260924 11:40:50.203562 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae648c1f2624981b524889a108e7c7e" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25f75af54ca42198eb2aa89142a677e" is_pre_election: true
I20260924 11:40:50.203800 2286 raft_consensus.cc:2454] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 0.
I20260924 11:40:50.203816 2564 raft_consensus.cc:2454] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 0.
I20260924 11:40:50.204252 2351 leader_election.cc:302] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99e479f763114cfb9616d6f783e4e380, c25f75af54ca42198eb2aa89142a677e; no voters:
I20260924 11:40:50.204499 2730 raft_consensus.cc:2781] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:40:50.204631 2730 raft_consensus.cc:482] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:40:50.204727 2730 raft_consensus.cc:3037] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:50.207646 2730 raft_consensus.cc:504] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:50.208261 2730 leader_election.cc:288] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 election: Requested vote from peers c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719), dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625)
I20260924 11:40:50.208822 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae648c1f2624981b524889a108e7c7e" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad4548aad6f473985d19a5134fe0378"
I20260924 11:40:50.209028 2564 raft_consensus.cc:3037] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:50.209092 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae648c1f2624981b524889a108e7c7e" candidate_uuid: "99e479f763114cfb9616d6f783e4e380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25f75af54ca42198eb2aa89142a677e"
I20260924 11:40:50.209285 2286 raft_consensus.cc:3037] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:50.212394 2564 raft_consensus.cc:2454] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 1.
I20260924 11:40:50.212448 2286 raft_consensus.cc:2454] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e479f763114cfb9616d6f783e4e380 in term 1.
I20260924 11:40:50.212827 2351 leader_election.cc:302] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99e479f763114cfb9616d6f783e4e380, c25f75af54ca42198eb2aa89142a677e; no voters:
I20260924 11:40:50.213075 2730 raft_consensus.cc:2781] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:50.213456 2730 raft_consensus.cc:686] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 1 LEADER]: Becoming Leader. State: Replica: 99e479f763114cfb9616d6f783e4e380, State: Running, Role: LEADER
I20260924 11:40:50.213716 2730 consensus_queue.cc:237] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } }
I20260924 11:40:50.216260 2124 catalog_manager.cc:5495] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e479f763114cfb9616d6f783e4e380 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "99e479f763114cfb9616d6f783e4e380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e479f763114cfb9616d6f783e4e380" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 } health_report { overall_health: UNKNOWN } } }
I20260924 11:40:50.243382 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:50.238915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.0.97.2:37889)
0924 11:40:50.239394 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:50.239398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:50.239415 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:40:50.239456 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:50.239458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:50.239507 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:50.239580 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:40:50.239845 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:50.240579 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:50.242056 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:50.242406 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:50.242486 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:50.242824 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:40:50.242896 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:40:50.242949 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:50.243057 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 11:40:50.243103 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":291,"threads_started":1}
I20260924 11:40:50.251010 2564 raft_consensus.cc:1261] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Refusing update from remote peer 99e479f763114cfb9616d6f783e4e380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:50.251120 2286 raft_consensus.cc:1261] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Refusing update from remote peer 99e479f763114cfb9616d6f783e4e380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:40:50.251569 2752 consensus_queue.cc:1035] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad4548aad6f473985d19a5134fe0378" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:40:50.251837 2730 consensus_queue.cc:1035] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25f75af54ca42198eb2aa89142a677e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:40:50.268522 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 11:40:51.856292 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:51.856473 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:40:51.859637 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11130bf92230466aacede0df37c45663: Sending DeleteTablet for 3 replicas of tablet 5ae648c1f2624981b524889a108e7c7e
I20260924 11:40:51.860241 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae648c1f2624981b524889a108e7c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46568
I20260924 11:40:51.860314 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae648c1f2624981b524889a108e7c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260924 11:40:51.860356 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae648c1f2624981b524889a108e7c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:40:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260924 11:40:51.861042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2196
I20260924 11:40:51.862710 2785 tablet_replica.cc:330] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: stopping tablet replica
I20260924 11:40:51.862941 2785 raft_consensus.cc:2229] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:40:51.863339 2785 raft_consensus.cc:2258] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:51.864468 2785 ts_tablet_manager.cc:1905] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:51.871856 2786 tablet_replica.cc:330] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: stopping tablet replica
I20260924 11:40:51.872082 2786 raft_consensus.cc:2229] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:51.872330 2786 raft_consensus.cc:2258] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:51.874181 2786 ts_tablet_manager.cc:1905] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:51.874320 2785 ts_tablet_manager.cc:1918] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:40:51.874475 2785 log.cc:1196] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/5ae648c1f2624981b524889a108e7c7e
I20260924 11:40:51.874992 2785 ts_tablet_manager.cc:1939] T 5ae648c1f2624981b524889a108e7c7e P 99e479f763114cfb9616d6f783e4e380: Deleting consensus metadata
I20260924 11:40:51.876183 2110 catalog_manager.cc:4841] TS 99e479f763114cfb9616d6f783e4e380 (127.0.97.2:37889): tablet 5ae648c1f2624981b524889a108e7c7e (table test-table [id=d6f5d365dcb24af9930d9907d500fb74]) successfully deleted
I20260924 11:40:51.878507 2784 tablet_replica.cc:330] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: stopping tablet replica
I20260924 11:40:51.878702 2784 raft_consensus.cc:2229] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:40:51.878916 2784 raft_consensus.cc:2258] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:40:51.879678 2784 ts_tablet_manager.cc:1905] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:40:51.880852 2786 ts_tablet_manager.cc:1918] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:40:51.880972 2786 log.cc:1196] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/5ae648c1f2624981b524889a108e7c7e
I20260924 11:40:51.881381 2786 ts_tablet_manager.cc:1939] T 5ae648c1f2624981b524889a108e7c7e P dad4548aad6f473985d19a5134fe0378: Deleting consensus metadata
I20260924 11:40:51.882259 2109 catalog_manager.cc:4841] TS dad4548aad6f473985d19a5134fe0378 (127.0.97.3:42625): tablet 5ae648c1f2624981b524889a108e7c7e (table test-table [id=d6f5d365dcb24af9930d9907d500fb74]) successfully deleted
I20260924 11:40:51.884946 2784 ts_tablet_manager.cc:1918] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:40:51.885067 2784 log.cc:1196] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/5ae648c1f2624981b524889a108e7c7e
I20260924 11:40:51.885419 2784 ts_tablet_manager.cc:1939] T 5ae648c1f2624981b524889a108e7c7e P c25f75af54ca42198eb2aa89142a677e: Deleting consensus metadata
I20260924 11:40:51.886148 2109 catalog_manager.cc:4841] TS c25f75af54ca42198eb2aa89142a677e (127.0.97.1:42719): tablet 5ae648c1f2624981b524889a108e7c7e (table test-table [id=d6f5d365dcb24af9930d9907d500fb74]) successfully deleted
Sep 24 11:40:51 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:40:51.900607 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:51.862238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.97.1:42719)
0924 11:40:51.877752 (+ 15514us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:51.877758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:51.877781 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:40:51.877836 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:51.877841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:51.877916 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:51.878025 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:51.879616 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:51.880484 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:51.881780 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:51.882333 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:51.882458 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:51.896073 (+ 13615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:51.896085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:51.896100 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:51.896129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:51.898631 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:51.899158 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:51.899165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:51.899167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:51.899275 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:51.899539 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:51.899542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:51.899560 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:51.899843 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:51.900025 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:51.900247 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:40:51.900392 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15378,"thread_start_us":15901,"threads_started":1}
W20260924 11:40:51.901240 2326 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56212
W20260924 11:40:51.901767 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:40:51.901924 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 2196
I20260924 11:40:51.927245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2335
Sep 24 11:40:51 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:40:51.940340 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:51.927809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.0.97.2:37889)
0924 11:40:51.928311 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:51.928315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:51.928331 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:40:51.928377 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:51.928380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:51.928434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:51.928516 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:51.929557 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:51.930305 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:51.931375 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:51.931739 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:51.931823 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:51.935424 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:51.935434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:51.935449 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:51.935479 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:51.938123 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:51.938616 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:51.938621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:51.938623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:51.938746 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:51.939017 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:51.939020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:51.939022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:51.939292 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:51.939498 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:51.939790 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 11:40:51.939933 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":233,"threads_started":1}
W20260924 11:40:51.940943 2465 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42530
W20260924 11:40:51.941493 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2335: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:40:51.941643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 2335
I20260924 11:40:51.962709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2474
Sep 24 11:40:51 dist-test-slave-qhrh krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:40:51.975773 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:51.963315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.0.97.3:42625)
0924 11:40:51.963743 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:51.963748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:51.963762 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:40:51.963807 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:51.963810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:51.963874 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:51.963956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:51.965114 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:51.965831 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:51.966963 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:51.967339 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:51.967425 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:51.971215 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:51.971226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:51.971241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:51.971266 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:51.973635 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:51.974169 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:51.974175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:51.974177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:51.974287 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:51.974572 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:51.974575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:51.974576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:51.974888 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:51.975069 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:51.975273 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 11:40:51.975435 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":239,"threads_started":1}
W20260924 11:40:51.976490 2604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308
W20260924 11:40:51.977042 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:40:51.977164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 2474
I20260924 11:40:51.997624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2093
I20260924 11:40:51.998540 2165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:52.104597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 2093
2026-09-24T11:40:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8567 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:52 dist-test-slave-qhrh krb5kdc[2794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:52 dist-test-slave-qhrh krb5kdc[2794](info): set up 2 sockets
Sep 24 11:40:52 dist-test-slave-qhrh krb5kdc[2794](info): commencing operation
krb5kdc: starting...
W20260924 11:40:53.136075 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:40:53 dist-test-slave-qhrh krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:40:53.301770 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41379
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:53.588831 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:53.589066 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:53.589221 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:53.595801 2807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:53.595932 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:53.595973 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:53.596052 2807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:53.596110 2807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:53.604781 2807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:53.605881 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:53.607121 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:53.618695 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:53.618736 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:53.618652 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:53.619246 2807 server_base.cc:1034] running on GCE node
I20260924 11:40:53.619951 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:53.621137 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:53.622323 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790250053622275 us; error 46 us; skew 500 ppm
I20260924 11:40:53.622576 2807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:53 dist-test-slave-qhrh krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:53.625358 2807 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:53.626680 2807 webserver.cc:460] Webserver started at http://127.0.97.62:44739/ using document root <none> and password file <none>
I20260924 11:40:53.627277 2807 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:53.627401 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:53.627710 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:53.630425 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "a5313b21f8fa442a89db43a5518d1566"
format_stamp: "Formatted at 2026-09-24 11:40:53 on dist-test-slave-qhrh"
server_key: "681289b57b03b3e98692ab027ffae596"
server_key_iv: "50bac5f8350c647bea4a270af304bfa7"
server_key_version: "encryptionkey@0"
I20260924 11:40:53.631181 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "a5313b21f8fa442a89db43a5518d1566"
format_stamp: "Formatted at 2026-09-24 11:40:53 on dist-test-slave-qhrh"
server_key: "681289b57b03b3e98692ab027ffae596"
server_key_iv: "50bac5f8350c647bea4a270af304bfa7"
server_key_version: "encryptionkey@0"
I20260924 11:40:53.635596 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260924 11:40:53.638685 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:53.639791 2807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:53.639942 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "a5313b21f8fa442a89db43a5518d1566"
format_stamp: "Formatted at 2026-09-24 11:40:53 on dist-test-slave-qhrh"
server_key: "681289b57b03b3e98692ab027ffae596"
server_key_iv: "50bac5f8350c647bea4a270af304bfa7"
server_key_version: "encryptionkey@0"
I20260924 11:40:53.640110 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:53.661369 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:53.664686 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:53.664960 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:53.681308 2807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41379
I20260924 11:40:53.681385 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41379 every 8 connection(s)
I20260924 11:40:53.682385 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:53.685748 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:40:53.685926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 2807
I20260924 11:40:53.686110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:53.686894 388 external_mini_cluster.cc:1428] Setting key 4238a39f512999c3acb8812855d0cfbc
I20260924 11:40:53.698177 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566: Bootstrap starting.
Sep 24 11:40:53 dist-test-slave-qhrh krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:53.701293 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:53.702333 2874 log.cc:824] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:53.704843 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566: No bootstrap required, opened a new log
I20260924 11:40:53.707867 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:53.688737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.0.97.62:41379)
0924 11:40:53.689391 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:53.689400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:53.689432 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:40:53.691247 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:53.691266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:53.691587 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:53.691870 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:53.693524 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:53.694742 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:53.696447 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:53.697034 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:53.697152 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:53.701118 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:53.701124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:53.701135 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:53.701174 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:53.704289 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:53.704963 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:53.704977 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:53.704983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:53.705175 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:53.705545 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:53.705549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:53.705551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:53.705959 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:53.706283 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:53.706711 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 11:40:53.707048 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":343,"threads_started":1}
I20260924 11:40:53.711843 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5313b21f8fa442a89db43a5518d1566" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41379 } }
I20260924 11:40:53.712193 2874 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:53.712304 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5313b21f8fa442a89db43a5518d1566, State: Initialized, Role: FOLLOWER
I20260924 11:40:53.712836 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5313b21f8fa442a89db43a5518d1566" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41379 } }
I20260924 11:40:53.713056 2874 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:53.713200 2874 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:53.713347 2874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:53.715363 2874 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5313b21f8fa442a89db43a5518d1566" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41379 } }
I20260924 11:40:53.715783 2874 leader_election.cc:302] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5313b21f8fa442a89db43a5518d1566; no voters:
I20260924 11:40:53.716339 2874 leader_election.cc:288] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:53.716461 2879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:53.717065 2879 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [term 1 LEADER]: Becoming Leader. State: Replica: a5313b21f8fa442a89db43a5518d1566, State: Running, Role: LEADER
I20260924 11:40:53.717540 2879 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5313b21f8fa442a89db43a5518d1566" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41379 } }
I20260924 11:40:53.718092 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:53.720453 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5313b21f8fa442a89db43a5518d1566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5313b21f8fa442a89db43a5518d1566" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41379 } } }
I20260924 11:40:53.720508 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5313b21f8fa442a89db43a5518d1566. Latest consensus state: current_term: 1 leader_uuid: "a5313b21f8fa442a89db43a5518d1566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5313b21f8fa442a89db43a5518d1566" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41379 } } }
I20260924 11:40:53.720815 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:53.720814 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:53.722669 2887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:53.726681 2887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:53.734220 2887 catalog_manager.cc:1347] Generated new cluster ID: 2fa21379f0e74bb4aa3d9a7d6c3a43d9
I20260924 11:40:53.734318 2887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:53.752840 2887 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:53.753885 2887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:53.761950 2887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5313b21f8fa442a89db43a5518d1566: Generated new TSK 0
I20260924 11:40:53.762578 2887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:40:53.809571 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41379
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:54.105631 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:54.105887 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:54.105988 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:54.112389 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:54.112478 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:54.112578 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:40:54.121312 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:54.122383 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:54.123446 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:54.135594 2905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:54.135545 2908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:54.135557 2906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:54.136210 2900 server_base.cc:1034] running on GCE node
I20260924 11:40:54.136790 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:54.137514 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:54.138705 2900 hybrid_clock.cc:648] HybridClock initialized: now 1790250054138648 us; error 54 us; skew 500 ppm
I20260924 11:40:54.138927 2900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:54 dist-test-slave-qhrh krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:54.141836 2900 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:40:54.143118 2900 webserver.cc:460] Webserver started at http://127.0.97.1:35427/ using document root <none> and password file <none>
I20260924 11:40:54.143828 2900 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:54.143947 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:54.144292 2900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:54.147017 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "295332de00aa4fd3b7b05ea71cf9f213"
format_stamp: "Formatted at 2026-09-24 11:40:54 on dist-test-slave-qhrh"
server_key: "616598ebbbd78a893526158471f7d7ac"
server_key_iv: "7b76b3168195a01f2f3df03d8123bd60"
server_key_version: "encryptionkey@0"
I20260924 11:40:54.147742 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "295332de00aa4fd3b7b05ea71cf9f213"
format_stamp: "Formatted at 2026-09-24 11:40:54 on dist-test-slave-qhrh"
server_key: "616598ebbbd78a893526158471f7d7ac"
server_key_iv: "7b76b3168195a01f2f3df03d8123bd60"
server_key_version: "encryptionkey@0"
I20260924 11:40:54.152251 2900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260924 11:40:54.155237 2915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:54.156457 2900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:40:54.156574 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "295332de00aa4fd3b7b05ea71cf9f213"
format_stamp: "Formatted at 2026-09-24 11:40:54 on dist-test-slave-qhrh"
server_key: "616598ebbbd78a893526158471f7d7ac"
server_key_iv: "7b76b3168195a01f2f3df03d8123bd60"
server_key_version: "encryptionkey@0"
I20260924 11:40:54.156720 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:54.173703 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:54.177489 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:54.177686 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:54.178408 2900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:54.179440 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:54.179502 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:54.179574 2900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:54.179622 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:54.209818 2900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40475
I20260924 11:40:54.209892 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40475 every 8 connection(s)
I20260924 11:40:54.211033 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:40:54.213155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 2900
I20260924 11:40:54.213336 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:40:54.214154 388 external_mini_cluster.cc:1428] Setting key 4b4fb2c191fda0a31f0c3fae5bddfd86
Sep 24 11:40:54 dist-test-slave-qhrh krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:54.233047 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:54.213357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56595 (local address 127.0.97.62:41379)
0924 11:40:54.213809 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:54.213815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:54.215963 (+ 2148us) server_negotiation.cc:407] Connection header received
0924 11:40:54.217731 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:54.217739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:54.217821 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:54.217935 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:54.219369 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.220544 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.222085 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.222574 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.222737 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:54.226409 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:54.226418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:54.226425 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:54.226485 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:54.230040 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:54.230950 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:54.230953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:54.230955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:54.231059 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:54.231508 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:54.231511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:54.231513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:54.231792 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:54.231942 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:54.232695 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 11:40:54.232828 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:40:54.234439 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:54.213885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41379 (local address 127.0.97.1:56595)
0924 11:40:54.215794 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:54.215842 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:54.217491 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:54.218104 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:54.218112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:54.218682 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:54.219204 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:54.219219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.220682 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.220685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:54.221949 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:54.221959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.222777 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.222784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:54.223059 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:54.223711 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:54.223751 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:54.226202 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:54.230183 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:54.230193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:54.230210 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:54.230828 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:54.231165 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:54.231170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:54.231176 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:54.231406 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:54.231942 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:54.231951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:54.232347 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 11:40:54.232652 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 11:40:54.233106 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"thread_start_us":1390,"threads_started":1}
I20260924 11:40:54.235757 3028 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41379
I20260924 11:40:54.236017 3028 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 11:40:54.236755 3028 heartbeater.cc:511] Master 127.0.97.62:41379 requested a full tablet report, sending...
I20260924 11:40:54.238543 2839 ts_manager.cc:194] Registered new tserver with Master: 295332de00aa4fd3b7b05ea71cf9f213 (127.0.97.1:40475)
I20260924 11:40:54.240010 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:40:54.257016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41379
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:54.543314 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:54.543627 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:54.543784 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:54.550128 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:54.550244 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:54.550390 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:40:54.558529 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:54.559638 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:54.560802 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:54.572753 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:54.572789 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:54.572747 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:54.573518 3034 server_base.cc:1034] running on GCE node
I20260924 11:40:54.574079 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:54.574916 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:54.576125 3034 hybrid_clock.cc:648] HybridClock initialized: now 1790250054576088 us; error 53 us; skew 500 ppm
I20260924 11:40:54.576447 3034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:54 dist-test-slave-qhrh krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:54.579247 3034 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:40:54.580476 3034 webserver.cc:460] Webserver started at http://127.0.97.2:42925/ using document root <none> and password file <none>
I20260924 11:40:54.580989 3034 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:54.581073 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:54.581283 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:54.583591 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "731c3d6e9e0443c9afe73f6d8ada000d"
format_stamp: "Formatted at 2026-09-24 11:40:54 on dist-test-slave-qhrh"
server_key: "2a93f18ceac968d62c1f508a4ed4832e"
server_key_iv: "151d7bc91e999019c47087b3a7971cb8"
server_key_version: "encryptionkey@0"
I20260924 11:40:54.584327 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "731c3d6e9e0443c9afe73f6d8ada000d"
format_stamp: "Formatted at 2026-09-24 11:40:54 on dist-test-slave-qhrh"
server_key: "2a93f18ceac968d62c1f508a4ed4832e"
server_key_iv: "151d7bc91e999019c47087b3a7971cb8"
server_key_version: "encryptionkey@0"
I20260924 11:40:54.588560 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:40:54.591370 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:54.592504 3034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:40:54.592639 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "731c3d6e9e0443c9afe73f6d8ada000d"
format_stamp: "Formatted at 2026-09-24 11:40:54 on dist-test-slave-qhrh"
server_key: "2a93f18ceac968d62c1f508a4ed4832e"
server_key_iv: "151d7bc91e999019c47087b3a7971cb8"
server_key_version: "encryptionkey@0"
I20260924 11:40:54.592824 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:54.609530 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:54.613343 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:54.613618 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:54.614569 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:54.615775 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:54.615841 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:54.615990 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:54.616037 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:54.646544 3034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33465
I20260924 11:40:54.646685 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33465 every 8 connection(s)
I20260924 11:40:54.647648 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:40:54.650682 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3034
I20260924 11:40:54.650828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:40:54.651294 388 external_mini_cluster.cc:1428] Setting key 00b9dba6c0e342fc06357aa064fea904
Sep 24 11:40:54 dist-test-slave-qhrh krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:54.669204 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:54.649840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39947 (local address 127.0.97.62:41379)
0924 11:40:54.650009 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:54.650014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:54.651695 (+ 1681us) server_negotiation.cc:407] Connection header received
0924 11:40:54.653758 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:54.653762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:54.653838 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:54.653916 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:54.655870 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.656569 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.658774 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.659422 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.659547 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:54.662728 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:54.662735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:54.662742 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:54.662780 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:54.666179 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:54.666827 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:54.666835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:54.666841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:54.666989 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:54.667368 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:54.667374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:54.667381 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:54.667750 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:54.667909 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:54.668594 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 11:40:54.668831 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:40:54.669939 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:54.650298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41379 (local address 127.0.97.2:39947)
0924 11:40:54.651554 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:54.651598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:54.653510 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:54.654161 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:54.654183 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:54.654918 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:54.655703 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:54.655729 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.656701 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.656708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:54.658653 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:54.658663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:54.659604 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:54.659608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:54.659770 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:54.660375 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:54.660412 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:54.662516 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:54.666294 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:54.666300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:54.666310 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:54.666712 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:54.667105 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:54.667108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:54.667111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:54.667262 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:54.667863 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:54.667868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:54.668126 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 11:40:54.668495 (+ 369us) client_negotiation.cc:240] Negotiation successful
0924 11:40:54.668943 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":263,"threads_started":1}
I20260924 11:40:54.671136 3162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41379
I20260924 11:40:54.671348 3162 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:54.671923 3162 heartbeater.cc:511] Master 127.0.97.62:41379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 11:40:54.673224 2839 ts_manager.cc:194] Registered new tserver with Master: 731c3d6e9e0443c9afe73f6d8ada000d (127.0.97.2:33465)
I20260924 11:40:54.674018 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:40:54.692360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41379
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:54.983783 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:54.984025 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:54.984109 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:54.990509 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:54.990607 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:54.990705 3167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:40:54.999676 3167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:55.000619 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:55.001654 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:55.015079 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:55.015076 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:55.015076 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:55.016611 3167 server_base.cc:1034] running on GCE node
I20260924 11:40:55.017082 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:55.017830 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:55.019002 3167 hybrid_clock.cc:648] HybridClock initialized: now 1790250055018968 us; error 27 us; skew 500 ppm
I20260924 11:40:55.019245 3167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:55 dist-test-slave-qhrh krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:55.022171 3167 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:40:55.023483 3167 webserver.cc:460] Webserver started at http://127.0.97.3:45533/ using document root <none> and password file <none>
I20260924 11:40:55.024107 3167 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:55.024257 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:55.024504 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:55.027045 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "e3599b70f1f84138beba77a3bb03b83f"
format_stamp: "Formatted at 2026-09-24 11:40:55 on dist-test-slave-qhrh"
server_key: "8969c7751bbee780651572403129f8ff"
server_key_iv: "26bcffbfa45283fff7edbb5787ea7fcc"
server_key_version: "encryptionkey@0"
I20260924 11:40:55.027737 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e3599b70f1f84138beba77a3bb03b83f"
format_stamp: "Formatted at 2026-09-24 11:40:55 on dist-test-slave-qhrh"
server_key: "8969c7751bbee780651572403129f8ff"
server_key_iv: "26bcffbfa45283fff7edbb5787ea7fcc"
server_key_version: "encryptionkey@0"
I20260924 11:40:55.032133 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 11:40:55.035223 3182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:55.036417 3167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:40:55.036563 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "e3599b70f1f84138beba77a3bb03b83f"
format_stamp: "Formatted at 2026-09-24 11:40:55 on dist-test-slave-qhrh"
server_key: "8969c7751bbee780651572403129f8ff"
server_key_iv: "26bcffbfa45283fff7edbb5787ea7fcc"
server_key_version: "encryptionkey@0"
I20260924 11:40:55.036765 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:55.060907 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:55.064194 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:55.064441 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:55.065297 3167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:55.066442 3167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:55.066538 3167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:55.066671 3167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:55.066748 3167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:55.098345 3167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36581
I20260924 11:40:55.098389 3294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36581 every 8 connection(s)
I20260924 11:40:55.099412 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:40:55.105495 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3167
I20260924 11:40:55.105635 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:40:55.106058 388 external_mini_cluster.cc:1428] Setting key a343ed5f3194cdaa4f3f586a1b03d2d5
Sep 24 11:40:55 dist-test-slave-qhrh krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:55.117754 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:55.101550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43817 (local address 127.0.97.62:41379)
0924 11:40:55.101720 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:55.101724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:55.102720 (+ 996us) server_negotiation.cc:407] Connection header received
0924 11:40:55.104470 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:55.104474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:55.104536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:55.104619 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:55.106107 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:55.107107 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:55.109257 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:55.109649 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:55.109749 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:55.112771 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:55.112775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:55.112778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:55.112804 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:55.115269 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:55.115879 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:55.115881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:55.115883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:55.115982 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:55.116469 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:55.116472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:55.116474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:55.116697 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:55.116809 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:55.117318 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 11:40:55.117454 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:40:55.118556 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:55.101913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41379 (local address 127.0.97.3:43817)
0924 11:40:55.102530 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:55.102586 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:55.104239 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:55.104823 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:55.104830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:55.105316 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:55.105942 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:55.105961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:55.107259 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:55.107265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:55.109129 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:55.109147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:55.109831 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:55.109836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:55.110012 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:55.110576 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:55.110600 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:55.112545 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:55.115383 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:55.115389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:55.115398 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:55.115779 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:55.116212 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:55.116220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:55.116223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:55.116377 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:55.116811 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:55.116815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:55.117044 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 11:40:55.117231 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 11:40:55.117521 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":245,"threads_started":1}
I20260924 11:40:55.119867 3295 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41379
I20260924 11:40:55.120126 3295 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:55.120746 3295 heartbeater.cc:511] Master 127.0.97.62:41379 requested a full tablet report, sending...
I20260924 11:40:55.121987 2839 ts_manager.cc:194] Registered new tserver with Master: e3599b70f1f84138beba77a3bb03b83f (127.0.97.3:36581)
I20260924 11:40:55.122733 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43817
I20260924 11:40:55.132706 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:40:55.159492 3030 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:40:55.157394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.0.97.62:41379)
0924 11:40:55.157548 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:55.157552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:55.158025 (+ 473us) server_negotiation.cc:407] Connection header received
0924 11:40:55.158152 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:55.158155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:55.158210 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:55.158290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:55.159257 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48006: BlockingRecv error: recv got EOF from 127.0.0.1:48006 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:40:55.162597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2900
W20260924 11:40:55.165107 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2900: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40475: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:40:55.165288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 2900
I20260924 11:40:55.177453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3034
W20260924 11:40:55.179719 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3034: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:33465: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:40:55.179845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3034
I20260924 11:40:55.194192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3167
W20260924 11:40:55.196621 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3167: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36581: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:40:55.196763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3167
I20260924 11:40:55.210889 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 2807
I20260924 11:40:55.211768 2869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:55.306985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 2807
2026-09-24T11:40:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3200 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:55 dist-test-slave-qhrh krb5kdc[3312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:55 dist-test-slave-qhrh krb5kdc[3312](info): set up 2 sockets
Sep 24 11:40:55 dist-test-slave-qhrh krb5kdc[3312](info): commencing operation
krb5kdc: starting...
W20260924 11:40:56.344041 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:40:56 dist-test-slave-qhrh krb5kdc[3312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:40:56.503129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46137
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:56.792073 3325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:56.792337 3325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:56.792522 3325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:56.798976 3325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:56.799053 3325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:56.799110 3325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:56.799176 3325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:56.799259 3325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:56.807763 3325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:56.808806 3325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:56.809933 3325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:56.820740 3330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:56.820801 3331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:56.820899 3333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:56.821556 3325 server_base.cc:1034] running on GCE node
I20260924 11:40:56.822271 3325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:56.823447 3325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:56.824652 3325 hybrid_clock.cc:648] HybridClock initialized: now 1790250056824617 us; error 46 us; skew 500 ppm
I20260924 11:40:56.824967 3325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:56 dist-test-slave-qhrh krb5kdc[3312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:56.827841 3325 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:56.829236 3325 webserver.cc:460] Webserver started at http://127.0.97.62:36009/ using document root <none> and password file <none>
I20260924 11:40:56.829824 3325 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:56.829962 3325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:56.830238 3325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:56.832909 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "f8d9415e66544ff9aea91a8609ecb948"
format_stamp: "Formatted at 2026-09-24 11:40:56 on dist-test-slave-qhrh"
server_key: "08b1e01895590d191d9bf40b204dbfbc"
server_key_iv: "47c71366ed1399378fc0e69afcbd1a72"
server_key_version: "encryptionkey@0"
I20260924 11:40:56.833702 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8d9415e66544ff9aea91a8609ecb948"
format_stamp: "Formatted at 2026-09-24 11:40:56 on dist-test-slave-qhrh"
server_key: "08b1e01895590d191d9bf40b204dbfbc"
server_key_iv: "47c71366ed1399378fc0e69afcbd1a72"
server_key_version: "encryptionkey@0"
I20260924 11:40:56.838222 3325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:40:56.841162 3340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:56.842583 3325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:40:56.842696 3325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "f8d9415e66544ff9aea91a8609ecb948"
format_stamp: "Formatted at 2026-09-24 11:40:56 on dist-test-slave-qhrh"
server_key: "08b1e01895590d191d9bf40b204dbfbc"
server_key_iv: "47c71366ed1399378fc0e69afcbd1a72"
server_key_version: "encryptionkey@0"
I20260924 11:40:56.842829 3325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:56.854168 3325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:56.857445 3325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:56.857661 3325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:56.873425 3325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46137
I20260924 11:40:56.873445 3391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46137 every 8 connection(s)
I20260924 11:40:56.874536 3325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:56.875955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3325
I20260924 11:40:56.876194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:56.876925 388 external_mini_cluster.cc:1428] Setting key 229bca32bf73273337b1de210a679596
I20260924 11:40:56.878578 3392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:40:56 dist-test-slave-qhrh krb5kdc[3312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:56.890626 3392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948: Bootstrap starting.
I20260924 11:40:56.893723 3392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:56.894912 3392 log.cc:824] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:56.897814 3392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948: No bootstrap required, opened a new log
I20260924 11:40:56.897837 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:56.878536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.0.97.62:46137)
0924 11:40:56.879277 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:56.879290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:56.879334 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:40:56.881302 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:56.881323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:56.881746 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:56.882112 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:56.883266 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:56.885251 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:56.886450 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:56.887419 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:56.887600 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:56.890964 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:56.890970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:56.890981 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:56.891015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:56.893768 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:56.894555 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:56.894561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:56.894565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:56.894712 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:56.895132 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:56.895135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:56.895137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:56.895584 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:56.895762 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:56.896324 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 11:40:56.896664 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":329,"threads_started":1}
I20260924 11:40:56.901212 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:40:56.905086 3392 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d9415e66544ff9aea91a8609ecb948" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46137 } }
I20260924 11:40:56.905381 3392 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:56.905478 3392 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d9415e66544ff9aea91a8609ecb948, State: Initialized, Role: FOLLOWER
I20260924 11:40:56.905992 3392 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d9415e66544ff9aea91a8609ecb948" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46137 } }
I20260924 11:40:56.906203 3392 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:56.906322 3392 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:56.906495 3392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:56.908492 3392 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d9415e66544ff9aea91a8609ecb948" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46137 } }
I20260924 11:40:56.908924 3392 leader_election.cc:302] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8d9415e66544ff9aea91a8609ecb948; no voters:
I20260924 11:40:56.909452 3392 leader_election.cc:288] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:56.909569 3398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:56.911154 3392 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:56.915134 3398 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [term 1 LEADER]: Becoming Leader. State: Replica: f8d9415e66544ff9aea91a8609ecb948, State: Running, Role: LEADER
I20260924 11:40:56.915762 3398 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d9415e66544ff9aea91a8609ecb948" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46137 } }
I20260924 11:40:56.918505 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8d9415e66544ff9aea91a8609ecb948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d9415e66544ff9aea91a8609ecb948" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46137 } } }
I20260924 11:40:56.918816 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:56.919288 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8d9415e66544ff9aea91a8609ecb948. Latest consensus state: current_term: 1 leader_uuid: "f8d9415e66544ff9aea91a8609ecb948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d9415e66544ff9aea91a8609ecb948" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46137 } } }
I20260924 11:40:56.919548 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948 [sys.catalog]: This master's current role is: LEADER
W20260924 11:40:56.932238 3412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:40:56.932358 3412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:40:56.932467 3413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:56.934751 3413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:56.944028 3413 catalog_manager.cc:1347] Generated new cluster ID: 2fda269ced814331bc9c9ed9bfc973b1
I20260924 11:40:56.944166 3413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:56.958595 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:56.936697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.0.97.62:46137)
0924 11:40:56.936890 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:56.936895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:56.940726 (+ 3831us) server_negotiation.cc:407] Connection header received
0924 11:40:56.940974 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:56.940980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:56.941052 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:56.941128 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:56.942360 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:56.943101 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:56.944810 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:56.945471 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:56.945599 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:56.956970 (+ 11371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:56.956977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:56.956982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:40:56.957794 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:56.957848 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:40:56.957888 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:56.958019 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:56.958155 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 11:40:56.958245 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:40:56.966017 3413 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:56.967655 3413 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:56.980695 3413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8d9415e66544ff9aea91a8609ecb948: Generated new TSK 0
I20260924 11:40:56.981495 3413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:40:56.991158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3325
I20260924 11:40:56.992817 3387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:40:57.082615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3325
2026-09-24T11:40:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1770 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:40:57 dist-test-slave-qhrh krb5kdc[3426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:40:57 dist-test-slave-qhrh krb5kdc[3426](info): set up 2 sockets
Sep 24 11:40:57 dist-test-slave-qhrh krb5kdc[3426](info): commencing operation
krb5kdc: starting...
W20260924 11:40:58.112088 388 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:40:58 dist-test-slave-qhrh krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:40:58.279448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44337
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:58.583936 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:58.584201 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:58.584394 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:58.591212 3439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:40:58.591325 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:58.591482 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:58.591584 3439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:40:58.591668 3439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:40:58.600751 3439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:58.601863 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:58.603089 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:58.614398 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:58.614462 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:58.614786 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:58.615127 3439 server_base.cc:1034] running on GCE node
I20260924 11:40:58.615819 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:58.617050 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:58.618283 3439 hybrid_clock.cc:648] HybridClock initialized: now 1790250058618232 us; error 47 us; skew 500 ppm
I20260924 11:40:58.618584 3439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:58 dist-test-slave-qhrh krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:58.621486 3439 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:40:58.622861 3439 webserver.cc:460] Webserver started at http://127.0.97.62:38195/ using document root <none> and password file <none>
I20260924 11:40:58.623435 3439 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:58.623522 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:58.623734 3439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:58.626559 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "a62ac8ca580944828ad36afa039300d9"
format_stamp: "Formatted at 2026-09-24 11:40:58 on dist-test-slave-qhrh"
server_key: "6fa3ae86bede56da47fe82cd5725397e"
server_key_iv: "33107f0fd077c463808b288b1e35eb0b"
server_key_version: "encryptionkey@0"
I20260924 11:40:58.627205 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "a62ac8ca580944828ad36afa039300d9"
format_stamp: "Formatted at 2026-09-24 11:40:58 on dist-test-slave-qhrh"
server_key: "6fa3ae86bede56da47fe82cd5725397e"
server_key_iv: "33107f0fd077c463808b288b1e35eb0b"
server_key_version: "encryptionkey@0"
I20260924 11:40:58.631803 3439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 11:40:58.634933 3454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:58.636080 3439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:40:58.636296 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "a62ac8ca580944828ad36afa039300d9"
format_stamp: "Formatted at 2026-09-24 11:40:58 on dist-test-slave-qhrh"
server_key: "6fa3ae86bede56da47fe82cd5725397e"
server_key_iv: "33107f0fd077c463808b288b1e35eb0b"
server_key_version: "encryptionkey@0"
I20260924 11:40:58.636461 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:58.660264 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:58.664032 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:58.664342 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:58.680896 3439 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44337
I20260924 11:40:58.680928 3505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44337 every 8 connection(s)
I20260924 11:40:58.681969 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:40:58.683259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3439
I20260924 11:40:58.683461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:40:58.684263 388 external_mini_cluster.cc:1428] Setting key 458984ac94f47cf06dd4a8e77d0f1354
I20260924 11:40:58.686138 3506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:40:58 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:58.698480 3506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9: Bootstrap starting.
I20260924 11:40:58.701310 3506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9: Neither blocks nor log segments found. Creating new log.
I20260924 11:40:58.702533 3506 log.cc:824] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:40:58.705058 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:58.686095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.0.97.62:44337)
0924 11:40:58.686937 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:58.686946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:58.686982 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:40:58.688433 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:58.688450 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:58.688748 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:58.689021 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:58.690552 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:58.691747 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:58.693353 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:58.694229 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:58.694377 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:58.698624 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:58.698634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:58.698655 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:58.698685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:58.701379 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:58.702106 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:58.702111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:58.702116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:58.702274 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:58.702697 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:58.702703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:58.702706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:58.703135 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:58.703449 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:58.703818 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 11:40:58.704200 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":405,"threads_started":1}
I20260924 11:40:58.705271 3506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9: No bootstrap required, opened a new log
I20260924 11:40:58.712061 3506 raft_consensus.cc:348] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62ac8ca580944828ad36afa039300d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44337 } }
I20260924 11:40:58.712433 3506 raft_consensus.cc:374] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:40:58.712550 3506 raft_consensus.cc:729] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62ac8ca580944828ad36afa039300d9, State: Initialized, Role: FOLLOWER
I20260924 11:40:58.713164 3506 consensus_queue.cc:260] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62ac8ca580944828ad36afa039300d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44337 } }
I20260924 11:40:58.713409 3506 raft_consensus.cc:388] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:40:58.713569 3506 raft_consensus.cc:482] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:40:58.713752 3506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:40:58.715691 3506 raft_consensus.cc:504] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62ac8ca580944828ad36afa039300d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44337 } }
I20260924 11:40:58.716197 3506 leader_election.cc:302] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a62ac8ca580944828ad36afa039300d9; no voters:
I20260924 11:40:58.717100 3506 leader_election.cc:288] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:40:58.717185 3511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:40:58.717729 3511 raft_consensus.cc:686] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [term 1 LEADER]: Becoming Leader. State: Replica: a62ac8ca580944828ad36afa039300d9, State: Running, Role: LEADER
I20260924 11:40:58.718263 3511 consensus_queue.cc:237] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62ac8ca580944828ad36afa039300d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44337 } }
I20260924 11:40:58.718933 3506 sys_catalog.cc:564] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:40:58.720711 3512 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a62ac8ca580944828ad36afa039300d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62ac8ca580944828ad36afa039300d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44337 } } }
I20260924 11:40:58.721026 3512 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:58.721721 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a62ac8ca580944828ad36afa039300d9. Latest consensus state: current_term: 1 leader_uuid: "a62ac8ca580944828ad36afa039300d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62ac8ca580944828ad36afa039300d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44337 } } }
I20260924 11:40:58.722054 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:40:58.725229 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:40:58.730346 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:40:58.737913 3516 catalog_manager.cc:1347] Generated new cluster ID: a58ea5961a2245048dea6095c3f9e4c5
I20260924 11:40:58.738000 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:40:58.766683 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:40:58.767835 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:40:58.779022 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a62ac8ca580944828ad36afa039300d9: Generated new TSK 0
I20260924 11:40:58.779675 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:40:58.827798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44337
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:59.125888 3532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:59.126156 3532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:59.126348 3532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:59.132908 3532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:59.132985 3532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:59.133153 3532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:40:59.142023 3532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:59.143100 3532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:59.144512 3532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:59.156461 3537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:59.156551 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:59.156512 3540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:59.157375 3532 server_base.cc:1034] running on GCE node
I20260924 11:40:59.157879 3532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:59.158623 3532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:59.159821 3532 hybrid_clock.cc:648] HybridClock initialized: now 1790250059159785 us; error 30 us; skew 500 ppm
I20260924 11:40:59.160199 3532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:59 dist-test-slave-qhrh krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:59.162997 3532 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:40:59.164355 3532 webserver.cc:460] Webserver started at http://127.0.97.1:35473/ using document root <none> and password file <none>
I20260924 11:40:59.164987 3532 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:59.165119 3532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:59.165390 3532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:59.168190 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "dc7b03bdfb4646d6923f92fc0f268e2f"
format_stamp: "Formatted at 2026-09-24 11:40:59 on dist-test-slave-qhrh"
server_key: "0012ea1779b644e3ccfe9f67868646c6"
server_key_iv: "2e2aeb61adca57711f5d87c76a8ee180"
server_key_version: "encryptionkey@0"
I20260924 11:40:59.168931 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dc7b03bdfb4646d6923f92fc0f268e2f"
format_stamp: "Formatted at 2026-09-24 11:40:59 on dist-test-slave-qhrh"
server_key: "0012ea1779b644e3ccfe9f67868646c6"
server_key_iv: "2e2aeb61adca57711f5d87c76a8ee180"
server_key_version: "encryptionkey@0"
I20260924 11:40:59.173377 3532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 11:40:59.176453 3547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:59.177735 3532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:40:59.177923 3532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "dc7b03bdfb4646d6923f92fc0f268e2f"
format_stamp: "Formatted at 2026-09-24 11:40:59 on dist-test-slave-qhrh"
server_key: "0012ea1779b644e3ccfe9f67868646c6"
server_key_iv: "2e2aeb61adca57711f5d87c76a8ee180"
server_key_version: "encryptionkey@0"
I20260924 11:40:59.178118 3532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:59.195063 3532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:59.198400 3532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:59.198670 3532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:59.199429 3532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:59.200618 3532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:59.200716 3532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:59.200847 3532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:59.200913 3532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:59.231635 3532 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42365
I20260924 11:40:59.231704 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42365 every 8 connection(s)
I20260924 11:40:59.232725 3532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:40:59.242502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3532
I20260924 11:40:59.242758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:40:59.243486 388 external_mini_cluster.cc:1428] Setting key 2a38c03d539c6ec9e6d4b54dacac6cec
Sep 24 11:40:59 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:59.250887 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:59.234834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42865 (local address 127.0.97.62:44337)
0924 11:40:59.235233 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:59.235237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:59.236066 (+ 829us) server_negotiation.cc:407] Connection header received
0924 11:40:59.237697 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:59.237702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:59.237786 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:59.237890 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:59.239344 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.240013 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.241369 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.241744 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.241860 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:59.245607 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:59.245611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:59.245614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:59.245639 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:59.248024 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:59.248782 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:59.248785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:59.248787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:59.248902 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:59.249309 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:59.249312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:59.249313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:59.249524 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:59.249619 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:59.250305 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 11:40:59.250511 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":223,"threads_started":1}
I20260924 11:40:59.251307 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:59.235284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44337 (local address 127.0.97.1:42865)
0924 11:40:59.235929 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:59.235962 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:59.237458 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:59.238098 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:59.238105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:59.238618 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:59.239146 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:59.239162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.240190 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.240195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:59.241245 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:59.241255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.241930 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.241938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:59.242223 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:59.242910 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:59.242943 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:59.245387 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:59.248164 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:59.248173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:59.248187 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:59.248678 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:59.248996 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:59.249003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:59.249008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:59.249204 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:59.249635 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:59.249644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:59.250003 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 11:40:59.250181 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 11:40:59.250452 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260924 11:40:59.252592 3660 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44337
I20260924 11:40:59.252871 3660 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:59.253615 3660 heartbeater.cc:511] Master 127.0.97.62:44337 requested a full tablet report, sending...
I20260924 11:40:59.255774 3471 ts_manager.cc:194] Registered new tserver with Master: dc7b03bdfb4646d6923f92fc0f268e2f (127.0.97.1:42365)
I20260924 11:40:59.257761 3471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42865
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:40:59.282862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44337
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:40:59.571406 3666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:40:59.571637 3666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:40:59.571718 3666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:40:59.578380 3666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:40:59.578460 3666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:40:59.578552 3666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:40:59.586962 3666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:40:59.587900 3666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:40:59.588955 3666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:40:59.600425 3671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:59.600474 3672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:40:59.600468 3674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:40:59.601446 3666 server_base.cc:1034] running on GCE node
I20260924 11:40:59.601994 3666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:40:59.602792 3666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:40:59.604022 3666 hybrid_clock.cc:648] HybridClock initialized: now 1790250059603980 us; error 53 us; skew 500 ppm
I20260924 11:40:59.604333 3666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:40:59 dist-test-slave-qhrh krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:40:59.607153 3666 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:40:59.608482 3666 webserver.cc:460] Webserver started at http://127.0.97.2:37293/ using document root <none> and password file <none>
I20260924 11:40:59.609052 3666 fs_manager.cc:359] Metadata directory not provided
I20260924 11:40:59.609139 3666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:40:59.609354 3666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:40:59.611771 3666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "c7074ba1888247328c8d24d37fe50ae0"
format_stamp: "Formatted at 2026-09-24 11:40:59 on dist-test-slave-qhrh"
server_key: "9d471cb1be36347fb81fae46dd8f3ec0"
server_key_iv: "7abe74e5eec9e9cb0f541b5f1e90f540"
server_key_version: "encryptionkey@0"
I20260924 11:40:59.612481 3666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c7074ba1888247328c8d24d37fe50ae0"
format_stamp: "Formatted at 2026-09-24 11:40:59 on dist-test-slave-qhrh"
server_key: "9d471cb1be36347fb81fae46dd8f3ec0"
server_key_iv: "7abe74e5eec9e9cb0f541b5f1e90f540"
server_key_version: "encryptionkey@0"
I20260924 11:40:59.616894 3666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:40:59.619863 3681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:59.621047 3666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:40:59.621192 3666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "c7074ba1888247328c8d24d37fe50ae0"
format_stamp: "Formatted at 2026-09-24 11:40:59 on dist-test-slave-qhrh"
server_key: "9d471cb1be36347fb81fae46dd8f3ec0"
server_key_iv: "7abe74e5eec9e9cb0f541b5f1e90f540"
server_key_version: "encryptionkey@0"
I20260924 11:40:59.621363 3666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:40:59.636705 3666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:40:59.640132 3666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:40:59.640471 3666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:40:59.641265 3666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:40:59.642366 3666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:40:59.642431 3666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:59.642544 3666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:40:59.642603 3666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:40:59.672978 3666 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38605
I20260924 11:40:59.673050 3793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38605 every 8 connection(s)
I20260924 11:40:59.674074 3666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:40:59.676556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3666
I20260924 11:40:59.676726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:40:59.677268 388 external_mini_cluster.cc:1428] Setting key b76d369b941c1e559235846cf7a514ea
Sep 24 11:40:59 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:40:59.695046 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:59.676344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57323 (local address 127.0.97.62:44337)
0924 11:40:59.676500 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:40:59.676504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:40:59.677823 (+ 1319us) server_negotiation.cc:407] Connection header received
0924 11:40:59.679727 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:40:59.679733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:40:59.679812 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:40:59.679917 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:40:59.681426 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.682214 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.683662 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.684072 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.684200 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:59.688253 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:40:59.688260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:40:59.688267 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:40:59.688304 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:40:59.691770 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:59.692584 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:59.692590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:59.692594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:59.692752 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:40:59.693176 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:40:59.693182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:40:59.693187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:40:59.693542 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:40:59.693692 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:40:59.694398 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 11:40:59.694644 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:40:59.695850 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:40:59.676936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44337 (local address 127.0.97.2:57323)
0924 11:40:59.677661 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 11:40:59.677706 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:40:59.679496 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:40:59.680125 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:40:59.680132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:40:59.680685 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:40:59.681284 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:59.681301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.682385 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.682392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:59.683528 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:40:59.683539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:40:59.684280 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:40:59.684291 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:40:59.684536 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:40:59.685245 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:40:59.685281 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:40:59.688050 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:40:59.691895 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:59.691903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:59.691915 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:59.692452 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:59.692861 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:40:59.692866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:40:59.692869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:40:59.693054 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:40:59.693676 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:40:59.693683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:40:59.693995 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 11:40:59.694254 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:40:59.694724 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":333,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:40:59.697182 3794 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44337
I20260924 11:40:59.697485 3794 heartbeater.cc:464] Registering TS with master...
I20260924 11:40:59.698170 3794 heartbeater.cc:511] Master 127.0.97.62:44337 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 11:40:59.699572 3471 ts_manager.cc:194] Registered new tserver with Master: c7074ba1888247328c8d24d37fe50ae0 (127.0.97.2:38605)
I20260924 11:40:59.700356 3471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:40:59.719355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44337
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:00.011400 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:00.011653 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:00.011744 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:00.018867 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:00.018975 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:00.019131 3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:00.028651 3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:00.029850 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:00.031049 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:00.043452 3805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:00.043506 3804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:00.043485 3807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:00.044760 3799 server_base.cc:1034] running on GCE node
I20260924 11:41:00.045279 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:00.046063 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:00.047255 3799 hybrid_clock.cc:648] HybridClock initialized: now 1790250060047214 us; error 24 us; skew 500 ppm
I20260924 11:41:00.047503 3799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:00.050388 3799 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:41:00.051573 3799 webserver.cc:460] Webserver started at http://127.0.97.3:38417/ using document root <none> and password file <none>
I20260924 11:41:00.052299 3799 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:00.052400 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:00.052620 3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:00.055250 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "33947b8c918c4f2cb21fcc6ed13d5737"
format_stamp: "Formatted at 2026-09-24 11:41:00 on dist-test-slave-qhrh"
server_key: "0587dca1c475c5761a462a7df468642d"
server_key_iv: "80bce7e90a5de0804b67890dbc236cc4"
server_key_version: "encryptionkey@0"
I20260924 11:41:00.055953 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "33947b8c918c4f2cb21fcc6ed13d5737"
format_stamp: "Formatted at 2026-09-24 11:41:00 on dist-test-slave-qhrh"
server_key: "0587dca1c475c5761a462a7df468642d"
server_key_iv: "80bce7e90a5de0804b67890dbc236cc4"
server_key_version: "encryptionkey@0"
I20260924 11:41:00.060622 3799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:41:00.063716 3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:00.065001 3799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:00.065119 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "33947b8c918c4f2cb21fcc6ed13d5737"
format_stamp: "Formatted at 2026-09-24 11:41:00 on dist-test-slave-qhrh"
server_key: "0587dca1c475c5761a462a7df468642d"
server_key_iv: "80bce7e90a5de0804b67890dbc236cc4"
server_key_version: "encryptionkey@0"
I20260924 11:41:00.065271 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:00.083478 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:00.087342 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:00.087563 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:00.088385 3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:00.089515 3799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:00.089579 3799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:00.089651 3799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:00.089690 3799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:00.120361 3799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41969
I20260924 11:41:00.120440 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41969 every 8 connection(s)
I20260924 11:41:00.121495 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:00.123144 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3799
I20260924 11:41:00.123349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:00.123884 388 external_mini_cluster.cc:1428] Setting key 2fadf68bee5fef5c306c0057de424e07
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:00.140280 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:00.123883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51079 (local address 127.0.97.62:44337)
0924 11:41:00.124062 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:00.124068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:00.125601 (+ 1533us) server_negotiation.cc:407] Connection header received
0924 11:41:00.127566 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:00.127571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:00.127672 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:00.127795 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:00.129206 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.129907 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.131350 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.131760 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.131882 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:00.134896 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:00.134901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:00.134906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:00.134944 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:00.137216 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.138055 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.138059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.138061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.138206 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.138690 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.138694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.138696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.138934 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:00.139031 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:00.139610 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 11:41:00.139942 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:41:00.141049 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:00.124529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44337 (local address 127.0.97.3:51079)
0924 11:41:00.125439 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:00.125490 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:00.127331 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:00.128015 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:00.128023 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:00.128540 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:00.129058 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:00.129074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.130036 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.130041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:00.131205 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:00.131216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.131960 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.131968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:00.132247 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:00.132783 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:00.132807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:00.134712 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:00.137352 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:00.137357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:00.137367 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:00.137915 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:00.138333 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:00.138338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:00.138343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:00.138559 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:00.139034 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:00.139039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:00.139292 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 11:41:00.139515 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 11:41:00.139947 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":347,"threads_started":1}
I20260924 11:41:00.142161 3927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44337
I20260924 11:41:00.142412 3927 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:00.143074 3927 heartbeater.cc:511] Master 127.0.97.62:44337 requested a full tablet report, sending...
I20260924 11:41:00.144474 3471 ts_manager.cc:194] Registered new tserver with Master: 33947b8c918c4f2cb21fcc6ed13d5737 (127.0.97.3:41969)
I20260924 11:41:00.145212 3471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51079
I20260924 11:41:00.150521 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:00.193387 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:00.181380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.0.97.62:44337)
0924 11:41:00.181545 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:00.181549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:00.182045 (+ 496us) server_negotiation.cc:407] Connection header received
0924 11:41:00.182228 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:00.182231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:00.182286 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:00.182360 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:00.183570 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.184305 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.185461 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.185858 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.185937 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:00.189227 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:00.189231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:00.189234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:00.189259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:00.191381 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.191918 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.191922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.191924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.192017 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.192343 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.192346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.192348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.192623 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:00.192811 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:00.193031 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 11:41:00.193167 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 11:41:00.193857 3471 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59964
I20260924 11:41:00.195811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3532
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:00.209956 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:00.196940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.0.97.1:42365)
0924 11:41:00.197460 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:00.197467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:00.197490 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:41:00.197540 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:00.197547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:00.197736 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:00.197856 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:00.198805 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.199773 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.200959 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.201434 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.201520 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:00.204831 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:00.204840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:00.204852 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:00.204888 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:00.207356 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.208007 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.208014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.208016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.208130 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.208541 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.208547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.208551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.208850 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:00.209140 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:00.209395 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 11:41:00.209652 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":274,"threads_started":1}
W20260924 11:41:00.211050 3655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50124
W20260924 11:41:00.212194 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:41:00.212342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3532
I20260924 11:41:00.227618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3666
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:41:00.242620 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:00.228385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.97.2:38605)
0924 11:41:00.228870 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:00.228876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:00.228897 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:00.228953 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:00.228959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:00.229142 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:00.229317 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:00.230477 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.231463 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.232750 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.233298 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.233455 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:00.237424 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:00.237433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:00.237444 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:00.237474 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:00.239997 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.240706 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.240712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.240714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.240821 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.241176 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.241185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.241187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.241484 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:00.241773 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:00.242071 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 11:41:00.242317 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
W20260924 11:41:00.243741 3789 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49760
W20260924 11:41:00.244807 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3666: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:41:00.244992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3666
I20260924 11:41:00.259550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3799
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250060, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:41:00.273898 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:00.260344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.97.3:41969)
0924 11:41:00.260855 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:00.260861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:00.260887 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:41:00.260936 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:00.260941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:00.261118 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:00.261230 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:00.262305 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.263255 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.264467 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:00.264985 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:00.265077 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:00.268959 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:00.268969 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:00.268980 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:00.269007 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:00.271657 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.272282 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.272287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.272289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.272412 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:00.272699 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:00.272702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:00.272704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:00.272982 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:00.273184 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:00.273369 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 11:41:00.273533 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
W20260924 11:41:00.275019 3922 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52682
W20260924 11:41:00.276032 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3799: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:41:00.276216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3799
I20260924 11:41:00.290719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3439
I20260924 11:41:00.291757 3501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:00.386883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3439
2026-09-24T11:41:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3309 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3944](info): set up 2 sockets
Sep 24 11:41:00 dist-test-slave-qhrh krb5kdc[3944](info): commencing operation
krb5kdc: starting...
W20260924 11:41:01.444183 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:01 dist-test-slave-qhrh krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:41:01.612897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40141
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:01.907888 3957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:01.908123 3957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:01.908295 3957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:01.915021 3957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:01.915107 3957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:01.915150 3957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:01.915205 3957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:01.915241 3957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:01.924130 3957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:01.925276 3957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:01.926512 3957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:01.937403 3962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:01.937451 3963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:01.937623 3965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:01.938220 3957 server_base.cc:1034] running on GCE node
I20260924 11:41:01.938863 3957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:01.940024 3957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:01.941250 3957 hybrid_clock.cc:648] HybridClock initialized: now 1790250061941193 us; error 43 us; skew 500 ppm
I20260924 11:41:01.941565 3957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:01 dist-test-slave-qhrh krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:01.944294 3957 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:41:01.945595 3957 webserver.cc:460] Webserver started at http://127.0.97.62:33343/ using document root <none> and password file <none>
I20260924 11:41:01.946228 3957 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:01.946357 3957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:01.946682 3957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:01.949378 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "1746d6bd573b49c48b0761642593ef80"
format_stamp: "Formatted at 2026-09-24 11:41:01 on dist-test-slave-qhrh"
server_key: "d58687e0935d6c8f2a1de9d836c139d1"
server_key_iv: "4f449356cd5b2bc31f3b82a1e013d0cf"
server_key_version: "encryptionkey@0"
I20260924 11:41:01.950088 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "1746d6bd573b49c48b0761642593ef80"
format_stamp: "Formatted at 2026-09-24 11:41:01 on dist-test-slave-qhrh"
server_key: "d58687e0935d6c8f2a1de9d836c139d1"
server_key_iv: "4f449356cd5b2bc31f3b82a1e013d0cf"
server_key_version: "encryptionkey@0"
I20260924 11:41:01.954685 3957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:41:01.958681 3972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:01.959729 3957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:01.959889 3957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "1746d6bd573b49c48b0761642593ef80"
format_stamp: "Formatted at 2026-09-24 11:41:01 on dist-test-slave-qhrh"
server_key: "d58687e0935d6c8f2a1de9d836c139d1"
server_key_iv: "4f449356cd5b2bc31f3b82a1e013d0cf"
server_key_version: "encryptionkey@0"
I20260924 11:41:01.960076 3957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:01.993687 3957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:01.997272 3957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:01.997541 3957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:02.014335 3957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40141
I20260924 11:41:02.014374 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40141 every 8 connection(s)
I20260924 11:41:02.015555 3957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:02.017177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 3957
I20260924 11:41:02.017385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:02.018157 388 external_mini_cluster.cc:1428] Setting key ffacadcab97746a50037c3f21ceb13fb
I20260924 11:41:02.019902 4024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:41:02 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:02.032966 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80: Bootstrap starting.
I20260924 11:41:02.036873 4024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:02.038072 4024 log.cc:824] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:02.040591 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:02.020087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.97.62:40141)
0924 11:41:02.020828 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:02.020838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:02.020874 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:41:02.022321 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:02.022339 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:02.022664 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:02.022966 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:02.024620 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:02.025919 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:02.027622 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:02.028354 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:02.028488 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:02.032826 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:02.032837 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:02.032853 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:02.032895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:02.036599 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:02.037230 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:02.037243 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:02.037252 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:02.037461 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:02.037774 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:02.037783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:02.037788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:02.038445 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:02.038797 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:02.039114 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 11:41:02.039439 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":313,"threads_started":1}
I20260924 11:41:02.041123 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80: No bootstrap required, opened a new log
I20260924 11:41:02.048454 4024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1746d6bd573b49c48b0761642593ef80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40141 } }
I20260924 11:41:02.048872 4024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:02.048971 4024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1746d6bd573b49c48b0761642593ef80, State: Initialized, Role: FOLLOWER
I20260924 11:41:02.049459 4024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1746d6bd573b49c48b0761642593ef80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40141 } }
I20260924 11:41:02.049662 4024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:02.049803 4024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:02.049954 4024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:02.051731 4024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1746d6bd573b49c48b0761642593ef80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40141 } }
I20260924 11:41:02.052191 4024 leader_election.cc:302] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1746d6bd573b49c48b0761642593ef80; no voters:
I20260924 11:41:02.052762 4024 leader_election.cc:288] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:02.052909 4029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:02.054344 4029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [term 1 LEADER]: Becoming Leader. State: Replica: 1746d6bd573b49c48b0761642593ef80, State: Running, Role: LEADER
I20260924 11:41:02.054663 4024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:02.054863 4029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1746d6bd573b49c48b0761642593ef80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40141 } }
I20260924 11:41:02.057621 4030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1746d6bd573b49c48b0761642593ef80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1746d6bd573b49c48b0761642593ef80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40141 } } }
I20260924 11:41:02.057595 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1746d6bd573b49c48b0761642593ef80. Latest consensus state: current_term: 1 leader_uuid: "1746d6bd573b49c48b0761642593ef80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1746d6bd573b49c48b0761642593ef80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40141 } } }
I20260924 11:41:02.058029 4030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:02.058043 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:02.059532 4036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:02.064281 4036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:02.072211 4036 catalog_manager.cc:1347] Generated new cluster ID: 1b58772000c94d3ab492967cc26a4c3b
I20260924 11:41:02.072299 4036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:02.087081 4036 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:02.088755 4036 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:02.094529 4036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1746d6bd573b49c48b0761642593ef80: Generated new TSK 0
I20260924 11:41:02.095221 4036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:41:02.137538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40141
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:02.429630 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:02.429872 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:02.430015 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:02.436717 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:02.436830 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:02.437002 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:02.446094 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:02.447268 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:02.448508 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:02.460741 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:02.460687 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:02.460791 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:02.461707 4050 server_base.cc:1034] running on GCE node
I20260924 11:41:02.462347 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:02.463217 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:02.464423 4050 hybrid_clock.cc:648] HybridClock initialized: now 1790250062464372 us; error 43 us; skew 500 ppm
I20260924 11:41:02.464730 4050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:02 dist-test-slave-qhrh krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:02.467605 4050 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:41:02.469014 4050 webserver.cc:460] Webserver started at http://127.0.97.1:36293/ using document root <none> and password file <none>
I20260924 11:41:02.469640 4050 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:02.469771 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:02.470055 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:02.472630 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "adf970e7047d49d4979dc59604f1a489"
format_stamp: "Formatted at 2026-09-24 11:41:02 on dist-test-slave-qhrh"
server_key: "e6cfa2049f41876841fc02488d282636"
server_key_iv: "cb104767f9d634a2950340df0d4841c5"
server_key_version: "encryptionkey@0"
I20260924 11:41:02.473305 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "adf970e7047d49d4979dc59604f1a489"
format_stamp: "Formatted at 2026-09-24 11:41:02 on dist-test-slave-qhrh"
server_key: "e6cfa2049f41876841fc02488d282636"
server_key_iv: "cb104767f9d634a2950340df0d4841c5"
server_key_version: "encryptionkey@0"
I20260924 11:41:02.477824 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:41:02.480762 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:02.481873 4050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:02.482023 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "adf970e7047d49d4979dc59604f1a489"
format_stamp: "Formatted at 2026-09-24 11:41:02 on dist-test-slave-qhrh"
server_key: "e6cfa2049f41876841fc02488d282636"
server_key_iv: "cb104767f9d634a2950340df0d4841c5"
server_key_version: "encryptionkey@0"
I20260924 11:41:02.482189 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:02.502727 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:02.506789 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:02.507074 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:02.507962 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:02.509126 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:02.509223 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:02.509366 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:02.509434 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:02.540226 4050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40045
I20260924 11:41:02.540282 4177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40045 every 8 connection(s)
I20260924 11:41:02.541319 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:02.542196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4050
I20260924 11:41:02.542387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:02.543138 388 external_mini_cluster.cc:1428] Setting key cce5882eb56bad426bd62862a7020c1c
Sep 24 11:41:02 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:41:02.561305 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:02.543851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60697 (local address 127.0.97.62:40141)
0924 11:41:02.544323 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:02.544328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:02.545351 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 11:41:02.547134 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:02.547143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:02.547223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:02.547353 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:02.548725 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:02.549894 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:02.551552 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:02.551942 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:02.552033 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:02.555932 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:02.555939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:02.555944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:02.555978 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:02.558407 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:02.559120 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:02.559125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:02.559128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:02.559336 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:02.559738 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:02.559749 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:02.559751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:02.559992 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:02.560139 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:02.560756 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 11:41:02.560964 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":213,"threads_started":1}
I20260924 11:41:02.562192 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:02.544407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40141 (local address 127.0.97.1:60697)
0924 11:41:02.545192 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:02.545234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:02.546892 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:02.547530 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:02.547537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:02.548010 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:02.548566 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:02.548591 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:02.550030 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:02.550034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:02.551414 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:02.551431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:02.552160 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:02.552169 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:02.552464 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:02.553100 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:02.553130 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:02.555705 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:02.558536 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:02.558545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:02.558558 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:02.558962 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:02.559476 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:02.559479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:02.559481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:02.559626 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:02.560117 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:02.560124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:02.560438 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 11:41:02.560676 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 11:41:02.561107 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 11:41:02.563736 4178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40141
I20260924 11:41:02.564025 4178 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:02.564713 4178 heartbeater.cc:511] Master 127.0.97.62:40141 requested a full tablet report, sending...
I20260924 11:41:02.566597 3989 ts_manager.cc:194] Registered new tserver with Master: adf970e7047d49d4979dc59604f1a489 (127.0.97.1:40045)
I20260924 11:41:02.568504 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:41:02.585034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40141
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:02.873484 4184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:02.873750 4184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:02.873898 4184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:02.880652 4184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:02.880774 4184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:02.880939 4184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:02.889731 4184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:02.890712 4184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:02.891820 4184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:02.903591 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:02.903658 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:02.903620 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:02.904430 4184 server_base.cc:1034] running on GCE node
I20260924 11:41:02.904950 4184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:02.905733 4184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:02.906916 4184 hybrid_clock.cc:648] HybridClock initialized: now 1790250062906876 us; error 29 us; skew 500 ppm
I20260924 11:41:02.907191 4184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:02 dist-test-slave-qhrh krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:02.909924 4184 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:41:02.911303 4184 webserver.cc:460] Webserver started at http://127.0.97.2:36917/ using document root <none> and password file <none>
I20260924 11:41:02.911890 4184 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:02.912020 4184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:02.912359 4184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:02.914968 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "526aa55cb1154e8ebdd4f50cc695b176"
format_stamp: "Formatted at 2026-09-24 11:41:02 on dist-test-slave-qhrh"
server_key: "b0636b1124d35c89270ea7e9a5a80ca3"
server_key_iv: "c6288fb97aba5d879d0479260b231c7a"
server_key_version: "encryptionkey@0"
I20260924 11:41:02.915692 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "526aa55cb1154e8ebdd4f50cc695b176"
format_stamp: "Formatted at 2026-09-24 11:41:02 on dist-test-slave-qhrh"
server_key: "b0636b1124d35c89270ea7e9a5a80ca3"
server_key_iv: "c6288fb97aba5d879d0479260b231c7a"
server_key_version: "encryptionkey@0"
I20260924 11:41:02.920854 4184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:41:02.923904 4199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:02.925086 4184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:02.925251 4184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "526aa55cb1154e8ebdd4f50cc695b176"
format_stamp: "Formatted at 2026-09-24 11:41:02 on dist-test-slave-qhrh"
server_key: "b0636b1124d35c89270ea7e9a5a80ca3"
server_key_iv: "c6288fb97aba5d879d0479260b231c7a"
server_key_version: "encryptionkey@0"
I20260924 11:41:02.925441 4184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:02.957726 4184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:02.962287 4184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:02.962571 4184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:02.963433 4184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:02.964701 4184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:02.964771 4184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:02.964931 4184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:02.964989 4184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:02.993884 4184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46201
I20260924 11:41:02.993968 4311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46201 every 8 connection(s)
I20260924 11:41:02.994992 4184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:02.998939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4184
I20260924 11:41:02.999122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:02.999505 388 external_mini_cluster.cc:1428] Setting key 9a49413b0ef976a30d248dc38f822689
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:03.014621 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:02.997586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40141 (local address 127.0.97.2:33351)
0924 11:41:02.998189 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:02.998221 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:02.999819 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:03.000487 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:03.000495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:03.001011 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:03.001475 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:03.001489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.002364 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.002375 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:03.003424 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:03.003434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.004197 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.004204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:03.004413 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.005023 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:03.005047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:03.007045 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:03.010030 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:03.010040 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:03.010053 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:03.010631 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:03.011069 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:03.011074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:03.011079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:03.011346 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:03.011964 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:03.011972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:03.012430 (+ 458us) client_negotiation.cc:769] Sending connection context
0924 11:41:03.012711 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 11:41:03.013197 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":257,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:41:03.017890 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:02.997196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33351 (local address 127.0.97.62:40141)
0924 11:41:02.997350 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:02.997354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:02.998328 (+ 974us) server_negotiation.cc:407] Connection header received
0924 11:41:03.000052 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.000058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.000174 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.000281 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.001620 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.002225 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.003552 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.003958 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.004081 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.007247 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.007252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.007255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.007283 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.009873 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.010752 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.010760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.010766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.010928 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.011464 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.011472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.011478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.011840 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.011980 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.012891 (+ 911us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.013223 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:41:03.018862 4312 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40141
I20260924 11:41:03.019124 4312 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 11:41:03.019949 4312 heartbeater.cc:511] Master 127.0.97.62:40141 requested a full tablet report, sending...
I20260924 11:41:03.021492 3989 ts_manager.cc:194] Registered new tserver with Master: 526aa55cb1154e8ebdd4f50cc695b176 (127.0.97.2:46201)
I20260924 11:41:03.022734 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:41:03.043252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40141
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:03.338925 4317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:03.339136 4317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:03.339247 4317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:03.345671 4317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:03.345763 4317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:03.345871 4317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:03.354277 4317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:03.355348 4317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:03.356606 4317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:03.367908 4325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:03.367904 4323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:03.367945 4322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:03.368705 4317 server_base.cc:1034] running on GCE node
I20260924 11:41:03.369230 4317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:03.369971 4317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:03.371158 4317 hybrid_clock.cc:648] HybridClock initialized: now 1790250063371110 us; error 40 us; skew 500 ppm
I20260924 11:41:03.371392 4317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:03.374025 4317 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:41:03.375315 4317 webserver.cc:460] Webserver started at http://127.0.97.3:33603/ using document root <none> and password file <none>
I20260924 11:41:03.375931 4317 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:03.376044 4317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:03.376351 4317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:03.378829 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "a3b053e18eda42e6826fad4b1a044d03"
format_stamp: "Formatted at 2026-09-24 11:41:03 on dist-test-slave-qhrh"
server_key: "a2da1f67abf22a6aad579497eeaa188c"
server_key_iv: "d57b28121e2253bd359638f03f9f4b47"
server_key_version: "encryptionkey@0"
I20260924 11:41:03.379483 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a3b053e18eda42e6826fad4b1a044d03"
format_stamp: "Formatted at 2026-09-24 11:41:03 on dist-test-slave-qhrh"
server_key: "a2da1f67abf22a6aad579497eeaa188c"
server_key_iv: "d57b28121e2253bd359638f03f9f4b47"
server_key_version: "encryptionkey@0"
I20260924 11:41:03.383846 4317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 11:41:03.387097 4332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:03.388422 4317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:41:03.388581 4317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "a3b053e18eda42e6826fad4b1a044d03"
format_stamp: "Formatted at 2026-09-24 11:41:03 on dist-test-slave-qhrh"
server_key: "a2da1f67abf22a6aad579497eeaa188c"
server_key_iv: "d57b28121e2253bd359638f03f9f4b47"
server_key_version: "encryptionkey@0"
I20260924 11:41:03.388770 4317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:03.401986 4317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:03.405404 4317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:03.405668 4317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:03.406509 4317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:03.407684 4317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:03.407793 4317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:03.407938 4317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:03.408007 4317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:03.438517 4317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44797
I20260924 11:41:03.438582 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44797 every 8 connection(s)
I20260924 11:41:03.439564 4317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:03.447191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4317
I20260924 11:41:03.447340 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:03.447846 388 external_mini_cluster.cc:1428] Setting key 88f0354d81d80040877dbebdc48032a6
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:03.456705 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.441674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51837 (local address 127.0.97.62:40141)
0924 11:41:03.441820 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:03.441824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:03.442745 (+ 921us) server_negotiation.cc:407] Connection header received
0924 11:41:03.444250 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.444254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.444330 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.444408 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.445689 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.446376 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.448456 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.449014 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.449121 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.452215 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.452220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.452222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.452247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.454253 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.454813 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.454820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.454826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.454957 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.455340 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.455343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.455345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.455573 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.455717 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.456348 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.456493 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:41:03.457440 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.441993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40141 (local address 127.0.97.3:51837)
0924 11:41:03.442592 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:03.442628 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:03.444000 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:03.444606 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:03.444613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:03.445051 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:03.445551 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:03.445566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.446499 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.446520 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:03.448302 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:03.448316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.449213 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.449219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:03.449473 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.450103 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:03.450147 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:03.452014 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:03.454355 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:03.454360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:03.454368 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:03.454688 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:03.455048 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:03.455051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:03.455053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:03.455247 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:03.455668 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:03.455673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:03.455984 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 11:41:03.456245 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 11:41:03.456545 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":251,"threads_started":1}
I20260924 11:41:03.458588 4445 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40141
I20260924 11:41:03.458840 4445 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:03.459452 4445 heartbeater.cc:511] Master 127.0.97.62:40141 requested a full tablet report, sending...
I20260924 11:41:03.460741 3989 ts_manager.cc:194] Registered new tserver with Master: a3b053e18eda42e6826fad4b1a044d03 (127.0.97.3:44797)
I20260924 11:41:03.461642 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51837
I20260924 11:41:03.462604 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:03.505563 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.492463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.97.1:40045)
0924 11:41:03.492989 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:03.492996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:03.493020 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:41:03.493158 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.493164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.493358 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.493467 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.494646 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.495617 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.496795 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.497272 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.497370 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.500883 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.500893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.500905 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.500931 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.503349 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.503855 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.503861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.503863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.503966 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.504335 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.504338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.504340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.504608 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.504827 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.505018 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.505185 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:03.521405 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.509293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.97.62:40141)
0924 11:41:03.509481 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:03.509487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:03.509642 (+ 155us) server_negotiation.cc:407] Connection header received
0924 11:41:03.509822 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.509827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.509890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.509986 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.511041 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.511706 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.512943 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.513278 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.513396 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.516822 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.516833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.516837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.516865 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.519102 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.519668 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.519675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.519681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.519799 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.520123 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.520129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.520132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.520414 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.520660 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.520918 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.521118 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 11:41:03.521855 3989 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34904
I20260924 11:41:03.522645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4050
I20260924 11:41:03.534416 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.523377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.97.1:40045)
0924 11:41:03.523558 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:03.523565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:03.523822 (+ 257us) server_negotiation.cc:407] Connection header received
0924 11:41:03.523979 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.523988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.524036 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.524126 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.525206 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.525834 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.527018 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.527411 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.527536 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.529681 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.529686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.529689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.529754 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.532194 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.532751 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.532757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.532760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.532872 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.533207 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.533214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.533217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.533433 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.533654 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.533931 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.534138 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:41:03.535008 4173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:03.610244 4178 heartbeater.cc:503] Master 127.0.97.62:40141 was elected leader, sending a full tablet report...
I20260924 11:41:03.611004 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4050
W20260924 11:41:03.626484 3974 connection.cc:535] server connection from 127.0.97.1:60697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:41:03.626724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4184
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:41:03.641045 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.627598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.97.2:46201)
0924 11:41:03.628115 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:03.628122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:03.628185 (+ 63us) server_negotiation.cc:407] Connection header received
0924 11:41:03.628251 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.628258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.628502 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.628627 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.629823 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.630837 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.631961 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.632541 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.632632 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.636368 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.636378 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.636391 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.636423 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.638894 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.639447 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.639453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.639455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.639570 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.639836 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.639839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.639841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.640113 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.640368 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.640574 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.640741 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
I20260924 11:41:03.642189 4307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:03.719482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4184
I20260924 11:41:03.735311 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4317
Sep 24 11:41:03 dist-test-slave-qhrh krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:41:03.750066 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:03.736217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.0.97.3:44797)
0924 11:41:03.736829 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:03.736836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:03.736864 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:41:03.736918 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:03.736924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:03.737117 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:03.737286 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:03.738463 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.739475 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.740886 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:03.741419 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:03.741549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:03.745221 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:03.745231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:03.745243 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:03.745272 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:03.747951 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.748530 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.748536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.748538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.748635 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:03.748946 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:03.748949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:03.748951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:03.749233 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:03.749435 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:03.749627 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 11:41:03.749811 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":321,"threads_started":1}
I20260924 11:41:03.751247 4440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:03.827883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4317
I20260924 11:41:03.843097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 3957
I20260924 11:41:03.843959 4019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:03.938077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 3957
2026-09-24T11:41:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3551 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:04 dist-test-slave-qhrh krb5kdc[4464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:04 dist-test-slave-qhrh krb5kdc[4464](info): set up 2 sockets
Sep 24 11:41:04 dist-test-slave-qhrh krb5kdc[4464](info): commencing operation
krb5kdc: starting...
W20260924 11:41:04.960234 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:05 dist-test-slave-qhrh krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:41:05.126971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33039
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:05.416486 4477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:05.416736 4477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:05.416843 4477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:05.424324 4477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:05.424444 4477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:05.424490 4477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:05.424547 4477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:05.424614 4477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:05.434195 4477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:05.435199 4477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:05.436386 4477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:05.446669 4482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:05.446697 4483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:05.446723 4485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:05.447320 4477 server_base.cc:1034] running on GCE node
I20260924 11:41:05.447995 4477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:05.449182 4477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:05.450399 4477 hybrid_clock.cc:648] HybridClock initialized: now 1790250065450370 us; error 39 us; skew 500 ppm
I20260924 11:41:05.450714 4477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:05 dist-test-slave-qhrh krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:05.453429 4477 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:41:05.454443 4477 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:05.454520 4477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:05.454728 4477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:05.457352 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "5430774df5354bf8a82136c487f4d9d4"
format_stamp: "Formatted at 2026-09-24 11:41:05 on dist-test-slave-qhrh"
server_key: "4ab4882b43ef7276f309046fff707ac9"
server_key_iv: "e4a2a8ab0f5ada88a73045a8e0a31c3f"
server_key_version: "encryptionkey@0"
I20260924 11:41:05.458043 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "5430774df5354bf8a82136c487f4d9d4"
format_stamp: "Formatted at 2026-09-24 11:41:05 on dist-test-slave-qhrh"
server_key: "4ab4882b43ef7276f309046fff707ac9"
server_key_iv: "e4a2a8ab0f5ada88a73045a8e0a31c3f"
server_key_version: "encryptionkey@0"
I20260924 11:41:05.462384 4477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:41:05.465402 4491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:05.466531 4477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:05.466708 4477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "5430774df5354bf8a82136c487f4d9d4"
format_stamp: "Formatted at 2026-09-24 11:41:05 on dist-test-slave-qhrh"
server_key: "4ab4882b43ef7276f309046fff707ac9"
server_key_iv: "e4a2a8ab0f5ada88a73045a8e0a31c3f"
server_key_version: "encryptionkey@0"
I20260924 11:41:05.466910 4477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:05.489463 4477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:05.493194 4477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:05.493446 4477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:05.509717 4477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33039
I20260924 11:41:05.509735 4542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33039 every 8 connection(s)
I20260924 11:41:05.510526 4477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:05.513592 4543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:05.520681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4477
I20260924 11:41:05.520885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:05.521457 388 external_mini_cluster.cc:1428] Setting key 609ea20169c5585cd9232e45d55a50e3
I20260924 11:41:05.522420 4543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4: Bootstrap starting.
I20260924 11:41:05.525785 4543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:05.527177 4543 log.cc:824] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:05.529915 4543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4: No bootstrap required, opened a new log
Sep 24 11:41:05 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:05.539762 4543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430774df5354bf8a82136c487f4d9d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33039 } }
I20260924 11:41:05.540187 4543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:05.540294 4543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5430774df5354bf8a82136c487f4d9d4, State: Initialized, Role: FOLLOWER
I20260924 11:41:05.540843 4543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430774df5354bf8a82136c487f4d9d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33039 } }
I20260924 11:41:05.541065 4543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:05.541210 4543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:05.541385 4543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:05.544339 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:05.523118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.0.97.62:33039)
0924 11:41:05.523809 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:05.523824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:05.523859 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:41:05.525716 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:05.525741 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:05.526173 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:05.526560 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:05.528808 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:05.530072 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:05.531854 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:05.532539 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:05.532664 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:05.536821 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:05.536830 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:05.536845 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:05.536894 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:05.540507 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:05.541114 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:05.541124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:05.541132 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:05.541345 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:05.541717 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:05.541722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:05.541725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:05.542310 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:05.542613 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:05.542872 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 11:41:05.543241 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":322,"threads_started":1}
I20260924 11:41:05.544251 4543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430774df5354bf8a82136c487f4d9d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33039 } }
I20260924 11:41:05.544728 4543 leader_election.cc:302] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5430774df5354bf8a82136c487f4d9d4; no voters:
I20260924 11:41:05.545267 4543 leader_election.cc:288] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:05.545415 4548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:05.546895 4548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [term 1 LEADER]: Becoming Leader. State: Replica: 5430774df5354bf8a82136c487f4d9d4, State: Running, Role: LEADER
I20260924 11:41:05.547466 4543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:05.547470 4548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430774df5354bf8a82136c487f4d9d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33039 } }
I20260924 11:41:05.550343 4550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5430774df5354bf8a82136c487f4d9d4. Latest consensus state: current_term: 1 leader_uuid: "5430774df5354bf8a82136c487f4d9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430774df5354bf8a82136c487f4d9d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33039 } } }
I20260924 11:41:05.550714 4550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:05.551179 4549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5430774df5354bf8a82136c487f4d9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430774df5354bf8a82136c487f4d9d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33039 } } }
I20260924 11:41:05.551470 4549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:05.553308 4558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:05.556985 4558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:05.564203 4558 catalog_manager.cc:1347] Generated new cluster ID: eb9271840b3a455fa01067149421ae8f
I20260924 11:41:05.564301 4558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:05.600219 4558 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:05.601357 4558 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:05.609553 4558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4: Generated new TSK 0
I20260924 11:41:05.610172 4558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:41:05.651298 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33039
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:05.941813 4569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:05.942031 4569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:05.942174 4569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:05.949214 4569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:05.949317 4569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:05.949481 4569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:05.958214 4569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:05.959198 4569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:05.960400 4569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:05.972728 4575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:05.972787 4577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:05.973368 4574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:05.974077 4569 server_base.cc:1034] running on GCE node
I20260924 11:41:05.974648 4569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:05.975493 4569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:05.976696 4569 hybrid_clock.cc:648] HybridClock initialized: now 1790250065976646 us; error 46 us; skew 500 ppm
I20260924 11:41:05.976981 4569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:05 dist-test-slave-qhrh krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:05.979624 4569 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:41:05.980594 4569 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:05.980710 4569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:05.980983 4569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:05.983388 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "74c9c67712504a0b93962069c3ca4264"
format_stamp: "Formatted at 2026-09-24 11:41:05 on dist-test-slave-qhrh"
server_key: "70dd420f3bf671708fd5a037fb78b9e0"
server_key_iv: "d9ea49283b46167cf5efdc662cb5623a"
server_key_version: "encryptionkey@0"
I20260924 11:41:05.984097 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "74c9c67712504a0b93962069c3ca4264"
format_stamp: "Formatted at 2026-09-24 11:41:05 on dist-test-slave-qhrh"
server_key: "70dd420f3bf671708fd5a037fb78b9e0"
server_key_iv: "d9ea49283b46167cf5efdc662cb5623a"
server_key_version: "encryptionkey@0"
I20260924 11:41:05.988559 4569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:41:05.991478 4583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:05.992614 4569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:05.992766 4569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "74c9c67712504a0b93962069c3ca4264"
format_stamp: "Formatted at 2026-09-24 11:41:05 on dist-test-slave-qhrh"
server_key: "70dd420f3bf671708fd5a037fb78b9e0"
server_key_iv: "d9ea49283b46167cf5efdc662cb5623a"
server_key_version: "encryptionkey@0"
I20260924 11:41:05.992957 4569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:06.005116 4569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:06.008502 4569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:06.008766 4569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:06.009608 4569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:06.010727 4569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:06.010844 4569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.010980 4569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:06.011070 4569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.041119 4569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43893
I20260924 11:41:06.041189 4695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43893 every 8 connection(s)
I20260924 11:41:06.041852 4569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:06.046645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4569
I20260924 11:41:06.046829 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:06.047458 388 external_mini_cluster.cc:1428] Setting key 5af7682511dc5b5aa5ff8a1dd15293ca
Sep 24 11:41:06 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:06.059489 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.044033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35937 (local address 127.0.97.62:33039)
0924 11:41:06.044216 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:06.044221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:06.045204 (+ 983us) server_negotiation.cc:407] Connection header received
0924 11:41:06.046858 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:06.046864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:06.046944 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:06.047044 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:06.048508 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.049285 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.050710 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.051038 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.051147 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.054187 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:06.054192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:06.054195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:06.054221 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:06.056823 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.057661 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.057664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.057666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.057764 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.058165 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.058167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.058169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.058369 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:06.058467 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:06.059153 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 11:41:06.059283 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:41:06.060812 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.044480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33039 (local address 127.0.97.1:35937)
0924 11:41:06.045058 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:06.045101 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:06.046600 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:06.047257 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:06.047264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:06.047789 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:06.048373 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:06.048388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.049406 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.049414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:06.050578 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:06.050587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.051219 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.051224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:06.051436 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.051973 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:06.051994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:06.054003 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:06.056945 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:06.056955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:06.056970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:06.057523 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:06.057860 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:06.057867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:06.057872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:06.058068 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:06.058480 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:06.058489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:06.058836 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 11:41:06.059090 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 11:41:06.059528 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":225,"threads_started":1}
I20260924 11:41:06.062242 4696 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33039
I20260924 11:41:06.062484 4696 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:06.063059 4696 heartbeater.cc:511] Master 127.0.97.62:33039 requested a full tablet report, sending...
I20260924 11:41:06.064962 4508 ts_manager.cc:194] Registered new tserver with Master: 74c9c67712504a0b93962069c3ca4264 (127.0.97.1:43893)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:41:06.066440 4508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:41:06.087703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33039
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:06.376426 4701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:06.376652 4701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:06.376736 4701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:06.383374 4701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:06.383461 4701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:06.383554 4701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:06.391942 4701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:06.392962 4701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:06.394006 4701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:06.405525 4707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:06.405553 4709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:06.405591 4706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:06.406138 4701 server_base.cc:1034] running on GCE node
I20260924 11:41:06.406708 4701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:06.407449 4701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:06.408651 4701 hybrid_clock.cc:648] HybridClock initialized: now 1790250066408612 us; error 51 us; skew 500 ppm
I20260924 11:41:06.408921 4701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:06 dist-test-slave-qhrh krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:06.411684 4701 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:41:06.412782 4701 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:06.412906 4701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:06.413203 4701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:06.415746 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "db193e62c2e64ef6a862cbe8845ac33c"
format_stamp: "Formatted at 2026-09-24 11:41:06 on dist-test-slave-qhrh"
server_key: "46c003c579f87db0bcc064b44794e7fb"
server_key_iv: "ebea4581f2efb6d6462c47619f821862"
server_key_version: "encryptionkey@0"
I20260924 11:41:06.416553 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "db193e62c2e64ef6a862cbe8845ac33c"
format_stamp: "Formatted at 2026-09-24 11:41:06 on dist-test-slave-qhrh"
server_key: "46c003c579f87db0bcc064b44794e7fb"
server_key_iv: "ebea4581f2efb6d6462c47619f821862"
server_key_version: "encryptionkey@0"
I20260924 11:41:06.421017 4701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 11:41:06.423878 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.425052 4701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:06.425222 4701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "db193e62c2e64ef6a862cbe8845ac33c"
format_stamp: "Formatted at 2026-09-24 11:41:06 on dist-test-slave-qhrh"
server_key: "46c003c579f87db0bcc064b44794e7fb"
server_key_iv: "ebea4581f2efb6d6462c47619f821862"
server_key_version: "encryptionkey@0"
I20260924 11:41:06.425426 4701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:06.446620 4701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:06.450412 4701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:06.450651 4701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:06.451378 4701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:06.452423 4701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:06.452548 4701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.452694 4701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:06.452776 4701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.483476 4701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46713
I20260924 11:41:06.483551 4827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46713 every 8 connection(s)
I20260924 11:41:06.484228 4701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:06.491237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4701
I20260924 11:41:06.491394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:06.491916 388 external_mini_cluster.cc:1428] Setting key 6cea29ef53d2579a96ea4e9e6dbecdd1
Sep 24 11:41:06 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:06.502914 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.486353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56685 (local address 127.0.97.62:33039)
0924 11:41:06.486498 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:06.486511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:06.487514 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 11:41:06.489241 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:06.489246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:06.489325 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:06.489429 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:06.490736 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.491798 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.493359 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.493705 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.493839 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.497428 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:06.497433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:06.497435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:06.497462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:06.500117 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.500713 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.500720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.500726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.500890 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.501236 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.501243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.501247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.501592 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:06.501731 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:06.502252 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 11:41:06.502499 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:41:06.503376 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.486745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33039 (local address 127.0.97.2:56685)
0924 11:41:06.487355 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:06.487388 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:06.488961 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:06.489622 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:06.489629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:06.490050 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:06.490564 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:06.490594 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.491932 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.491935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:06.493202 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:06.493213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.493921 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.493928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:06.494152 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.494713 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:06.494732 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:06.497252 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:06.500247 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:06.500252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:06.500261 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:06.500600 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:06.500994 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:06.500997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:06.500999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:06.501133 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:06.501697 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:06.501701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:06.501946 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 11:41:06.502124 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 11:41:06.502414 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":243,"threads_started":1}
I20260924 11:41:06.504701 4828 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33039
I20260924 11:41:06.504988 4828 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:06.505699 4828 heartbeater.cc:511] Master 127.0.97.62:33039 requested a full tablet report, sending...
I20260924 11:41:06.507117 4508 ts_manager.cc:194] Registered new tserver with Master: db193e62c2e64ef6a862cbe8845ac33c (127.0.97.2:46713)
I20260924 11:41:06.507859 4508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56685
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:41:06.534955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33039
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:06.822850 4833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:06.823050 4833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:06.823129 4833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:06.829802 4833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:06.829916 4833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:06.830093 4833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:06.838867 4833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:06.839855 4833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:06.840941 4833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:06.852560 4839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:06.852593 4838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:06.852644 4841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:06.853235 4833 server_base.cc:1034] running on GCE node
I20260924 11:41:06.853739 4833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:06.854552 4833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:06.855758 4833 hybrid_clock.cc:648] HybridClock initialized: now 1790250066855727 us; error 39 us; skew 500 ppm
I20260924 11:41:06.856041 4833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:06 dist-test-slave-qhrh krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:06.858906 4833 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:41:06.859897 4833 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:06.860011 4833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:06.860332 4833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:06.862668 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6"
format_stamp: "Formatted at 2026-09-24 11:41:06 on dist-test-slave-qhrh"
server_key: "ea9e086c815a1c72fe768b34e86c4f48"
server_key_iv: "c84adca36877fa5e22bd461b765af13b"
server_key_version: "encryptionkey@0"
I20260924 11:41:06.863354 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6"
format_stamp: "Formatted at 2026-09-24 11:41:06 on dist-test-slave-qhrh"
server_key: "ea9e086c815a1c72fe768b34e86c4f48"
server_key_iv: "c84adca36877fa5e22bd461b765af13b"
server_key_version: "encryptionkey@0"
I20260924 11:41:06.867800 4833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 11:41:06.870899 4847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.872048 4833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:06.872215 4833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6"
format_stamp: "Formatted at 2026-09-24 11:41:06 on dist-test-slave-qhrh"
server_key: "ea9e086c815a1c72fe768b34e86c4f48"
server_key_iv: "c84adca36877fa5e22bd461b765af13b"
server_key_version: "encryptionkey@0"
I20260924 11:41:06.872407 4833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:06.914598 4833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:06.918624 4833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:06.918875 4833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:06.919697 4833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:06.920878 4833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:06.920996 4833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.921128 4833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:06.921214 4833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:06.951561 4833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37485
I20260924 11:41:06.951572 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37485 every 8 connection(s)
I20260924 11:41:06.952385 4833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:06.959153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 4833
I20260924 11:41:06.959309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:06.959699 388 external_mini_cluster.cc:1428] Setting key c0b42246ab703658d45ca11ec2466562
Sep 24 11:41:06 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:06.969769 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.954440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38169 (local address 127.0.97.62:33039)
0924 11:41:06.954611 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:06.954615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:06.955588 (+ 973us) server_negotiation.cc:407] Connection header received
0924 11:41:06.957371 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:06.957376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:06.957489 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:06.957599 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:06.958914 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.959671 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.961373 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.961840 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.961941 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.964920 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:06.964924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:06.964927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:06.964952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:06.967433 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.968048 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.968054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.968057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.968236 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.968565 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.968568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.968570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.968789 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:06.968899 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:06.969430 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 11:41:06.969579 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:41:06.970558 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.954809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33039 (local address 127.0.97.3:38169)
0924 11:41:06.955442 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:06.955481 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:06.957109 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:06.957802 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:06.957810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:06.958269 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:06.958744 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:06.958762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.959816 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.959824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:06.961227 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:06.961240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.962044 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.962050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:06.962297 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.962857 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:06.962876 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:06.964760 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:06.967533 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:06.967538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:06.967547 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:06.967932 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:06.968320 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:06.968323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:06.968325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:06.968473 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:06.968881 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:06.968885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:06.969133 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 11:41:06.969336 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 11:41:06.969639 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":259,"threads_started":1}
I20260924 11:41:06.971733 4960 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33039
I20260924 11:41:06.971949 4960 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:06.972628 4960 heartbeater.cc:511] Master 127.0.97.62:33039 requested a full tablet report, sending...
I20260924 11:41:06.974017 4508 ts_manager.cc:194] Registered new tserver with Master: 0cc256c0971c4ff7ad7d7f97bd9a13b6 (127.0.97.3:37485)
I20260924 11:41:06.975090 4508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38169
I20260924 11:41:06.975145 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:41:06.989959 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:06.979403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.0.97.62:33039)
0924 11:41:06.979536 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:06.979540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:06.980030 (+ 490us) server_negotiation.cc:407] Connection header received
0924 11:41:06.980201 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:06.980211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:06.980268 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:06.980333 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:06.981490 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.982046 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.983066 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:06.983401 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:06.983475 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:06.985412 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:06.985417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:06.985420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:06.985453 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:06.987891 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.988588 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.988591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.988593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.988689 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:06.988972 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:06.988975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:06.988976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:06.989205 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:06.989396 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:06.989596 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 11:41:06.989732 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:41:07.001921 4508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:07.008426 4508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:07.046072 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.021739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46713 (local address 127.0.0.1:34926)
0924 11:41:07.035628 (+ 13889us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:07.035644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:07.035747 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:07.036689 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:07.036700 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:07.036737 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:07.037196 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.037210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.039355 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.039361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.041964 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.041976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.045010 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.045015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.045258 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.045305 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 11:41:07.045424 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 11:41:07.045592 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13782,"thread_start_us":13029,"threads_started":1}
I20260924 11:41:07.046933 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.022099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.0.97.2:46713)
0924 11:41:07.035706 (+ 13607us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.035715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.035742 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:41:07.035889 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.035896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.036131 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.036361 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:07.037391 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.039197 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.042134 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.044573 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.044669 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.044847 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.046361 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.046552 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13454,"thread_start_us":13352,"threads_started":1}
I20260924 11:41:07.050539 4763 tablet_service.cc:1467] Processing CreateTablet for tablet 05d24fed29fa4ecdadde18bd6e9d3477 (DEFAULT_TABLE table=test-table [id=422514afb9b5484db0a81e66b5f19d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:07.051779 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d24fed29fa4ecdadde18bd6e9d3477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:07.054873 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.021621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.97.3:37485)
0924 11:41:07.041732 (+ 20111us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.041740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.041768 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:41:07.041850 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.041856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.042050 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.042229 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:07.048046 (+ 5817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.050010 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.052306 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.053599 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.053723 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.053962 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.054414 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.054571 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19953,"thread_start_us":27491,"threads_started":1}
I20260924 11:41:07.055202 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.021519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37485 (local address 127.0.0.1:46800)
0924 11:41:07.034435 (+ 12916us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:07.034468 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:07.034604 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:07.042212 (+ 7608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:07.042216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:07.042237 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:07.042620 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.042634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.050214 (+ 7580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.050219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.052112 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.052124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.053857 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.053866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.054173 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.054226 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:41:07.054350 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 11:41:07.054526 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12683,"thread_start_us":14302,"threads_started":1}
I20260924 11:41:07.061841 4895 tablet_service.cc:1467] Processing CreateTablet for tablet 05d24fed29fa4ecdadde18bd6e9d3477 (DEFAULT_TABLE table=test-table [id=422514afb9b5484db0a81e66b5f19d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:07.062907 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d24fed29fa4ecdadde18bd6e9d3477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:07.064208 4978 tablet_bootstrap.cc:492] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Bootstrap starting.
I20260924 11:41:07.064663 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.036124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43893 (local address 127.0.0.1:43214)
0924 11:41:07.047727 (+ 11603us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:07.047748 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:07.047868 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:07.058467 (+ 10599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:07.058471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:07.058498 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:07.058666 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.058677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.060503 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.060507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.062194 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.062205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.063964 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.063969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.064131 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.064162 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:41:07.064246 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 11:41:07.064386 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11471,"thread_start_us":22683,"threads_started":1}
I20260924 11:41:07.065555 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.036501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.0.97.1:43893)
0924 11:41:07.057858 (+ 21357us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.057869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.057903 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:41:07.058003 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.058012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.058254 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.058467 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:07.058838 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.060358 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.062438 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.063781 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.064934 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.065137 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.065200 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.065351 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21144,"thread_start_us":22450,"threads_started":1}
I20260924 11:41:07.067778 4978 tablet_bootstrap.cc:654] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:07.068081 4631 tablet_service.cc:1467] Processing CreateTablet for tablet 05d24fed29fa4ecdadde18bd6e9d3477 (DEFAULT_TABLE table=test-table [id=422514afb9b5484db0a81e66b5f19d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:07.068950 4978 log.cc:824] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:07.069155 4631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d24fed29fa4ecdadde18bd6e9d3477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:07.069685 4696 heartbeater.cc:503] Master 127.0.97.62:33039 was elected leader, sending a full tablet report...
I20260924 11:41:07.072921 4978 tablet_bootstrap.cc:492] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: No bootstrap required, opened a new log
I20260924 11:41:07.073155 4978 ts_tablet_manager.cc:1397] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:41:07.076987 4980 tablet_bootstrap.cc:492] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Bootstrap starting.
I20260924 11:41:07.080217 4980 tablet_bootstrap.cc:654] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:07.080456 4981 tablet_bootstrap.cc:492] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Bootstrap starting.
I20260924 11:41:07.081395 4980 log.cc:824] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:07.082091 4978 raft_consensus.cc:348] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.082468 4978 raft_consensus.cc:374] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:07.082554 4978 raft_consensus.cc:729] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db193e62c2e64ef6a862cbe8845ac33c, State: Initialized, Role: FOLLOWER
I20260924 11:41:07.083127 4978 consensus_queue.cc:260] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.083428 4981 tablet_bootstrap.cc:654] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:07.084705 4981 log.cc:824] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:07.087553 4828 heartbeater.cc:503] Master 127.0.97.62:33039 was elected leader, sending a full tablet report...
I20260924 11:41:07.088663 4978 ts_tablet_manager.cc:1428] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260924 11:41:07.093957 4980 tablet_bootstrap.cc:492] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: No bootstrap required, opened a new log
I20260924 11:41:07.094112 4981 tablet_bootstrap.cc:492] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: No bootstrap required, opened a new log
I20260924 11:41:07.094192 4980 ts_tablet_manager.cc:1397] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260924 11:41:07.094323 4981 ts_tablet_manager.cc:1397] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260924 11:41:07.103672 4981 raft_consensus.cc:348] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.103670 4980 raft_consensus.cc:348] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.104051 4981 raft_consensus.cc:374] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:07.104051 4980 raft_consensus.cc:374] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:07.104142 4980 raft_consensus.cc:729] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc256c0971c4ff7ad7d7f97bd9a13b6, State: Initialized, Role: FOLLOWER
I20260924 11:41:07.104167 4981 raft_consensus.cc:729] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c9c67712504a0b93962069c3ca4264, State: Initialized, Role: FOLLOWER
I20260924 11:41:07.104715 4980 consensus_queue.cc:260] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.104715 4981 consensus_queue.cc:260] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.105669 4960 heartbeater.cc:503] Master 127.0.97.62:33039 was elected leader, sending a full tablet report...
I20260924 11:41:07.106388 4980 ts_tablet_manager.cc:1428] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260924 11:41:07.106693 4981 ts_tablet_manager.cc:1428] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
W20260924 11:41:07.203804 4961 tablet.cc:2369] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:07.226126 4986 raft_consensus.cc:482] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:07.226403 4986 raft_consensus.cc:504] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.227819 4986 leader_election.cc:288] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db193e62c2e64ef6a862cbe8845ac33c (127.0.97.2:46713), 74c9c67712504a0b93962069c3ca4264 (127.0.97.1:43893)
I20260924 11:41:07.233744 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.228058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46713 (local address 127.0.97.3:34865)
0924 11:41:07.228378 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:07.228397 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:07.228533 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:07.228969 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:07.228972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:07.228993 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:07.229230 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.229241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.230435 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.230439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.231786 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.231797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.233046 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.233051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.233230 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.233263 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:41:07.233360 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:41:07.233492 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"spinlock_wait_cycles":24960}
I20260924 11:41:07.233749 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.228224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34865 (local address 127.0.97.2:46713)
0924 11:41:07.228399 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.228404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.228477 (+ 73us) server_negotiation.cc:407] Connection header received
0924 11:41:07.228740 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.228746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.228823 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.228923 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:07.229390 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.230297 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.231936 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.232873 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.232994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.233067 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.233391 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.233518 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:41:07.234308 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.228214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51435 (local address 127.0.97.1:43893)
0924 11:41:07.228425 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.228431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.229049 (+ 618us) server_negotiation.cc:407] Connection header received
0924 11:41:07.229218 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.229223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.229295 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.229389 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:07.229937 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.230741 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.232481 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.233380 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.233487 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.233572 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.233946 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.234091 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 11:41:07.234759 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.228194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43893 (local address 127.0.97.3:51435)
0924 11:41:07.228945 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:07.228961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:07.229091 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:07.229453 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:07.229459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:07.229485 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:07.229776 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.229787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.230880 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.230883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.232344 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:07.232356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.233570 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.233576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:07.233737 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.233781 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:41:07.234466 (+ 685us) client_negotiation.cc:240] Negotiation successful
0924 11:41:07.234558 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":397,"threads_started":1}
I20260924 11:41:07.234669 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d24fed29fa4ecdadde18bd6e9d3477" candidate_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db193e62c2e64ef6a862cbe8845ac33c" is_pre_election: true
I20260924 11:41:07.235131 4783 raft_consensus.cc:2454] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cc256c0971c4ff7ad7d7f97bd9a13b6 in term 0.
I20260924 11:41:07.235678 4651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d24fed29fa4ecdadde18bd6e9d3477" candidate_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c9c67712504a0b93962069c3ca4264" is_pre_election: true
I20260924 11:41:07.235741 4848 leader_election.cc:302] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc256c0971c4ff7ad7d7f97bd9a13b6, db193e62c2e64ef6a862cbe8845ac33c; no voters:
W20260924 11:41:07.235901 4829 tablet.cc:2369] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:07.236128 4986 raft_consensus.cc:2781] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:07.236126 4651 raft_consensus.cc:2454] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cc256c0971c4ff7ad7d7f97bd9a13b6 in term 0.
I20260924 11:41:07.236286 4986 raft_consensus.cc:482] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:07.236394 4986 raft_consensus.cc:3037] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:07.238421 4986 raft_consensus.cc:504] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.238963 4986 leader_election.cc:288] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [CANDIDATE]: Term 1 election: Requested vote from peers db193e62c2e64ef6a862cbe8845ac33c (127.0.97.2:46713), 74c9c67712504a0b93962069c3ca4264 (127.0.97.1:43893)
I20260924 11:41:07.239264 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d24fed29fa4ecdadde18bd6e9d3477" candidate_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db193e62c2e64ef6a862cbe8845ac33c"
I20260924 11:41:07.239331 4651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d24fed29fa4ecdadde18bd6e9d3477" candidate_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c9c67712504a0b93962069c3ca4264"
I20260924 11:41:07.239454 4783 raft_consensus.cc:3037] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:07.239574 4651 raft_consensus.cc:3037] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:07.241465 4783 raft_consensus.cc:2454] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cc256c0971c4ff7ad7d7f97bd9a13b6 in term 1.
I20260924 11:41:07.241515 4651 raft_consensus.cc:2454] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cc256c0971c4ff7ad7d7f97bd9a13b6 in term 1.
I20260924 11:41:07.241770 4848 leader_election.cc:302] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc256c0971c4ff7ad7d7f97bd9a13b6, db193e62c2e64ef6a862cbe8845ac33c; no voters:
I20260924 11:41:07.242009 4986 raft_consensus.cc:2781] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:07.242398 4986 raft_consensus.cc:686] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0cc256c0971c4ff7ad7d7f97bd9a13b6, State: Running, Role: LEADER
I20260924 11:41:07.242780 4986 consensus_queue.cc:237] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } }
I20260924 11:41:07.246850 4508 catalog_manager.cc:5495] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cc256c0971c4ff7ad7d7f97bd9a13b6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cc256c0971c4ff7ad7d7f97bd9a13b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 } health_report { overall_health: UNKNOWN } } }
I20260924 11:41:07.264982 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.260060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.0.97.3:37485)
0924 11:41:07.260283 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.260289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.260403 (+ 114us) server_negotiation.cc:407] Connection header received
0924 11:41:07.260611 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.260614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.260674 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.260808 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:07.261141 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.261822 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.263103 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.263476 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.263597 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.264031 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:07.264496 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:07.264655 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.264710 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.264773 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:41:07.271660 4651 raft_consensus.cc:1261] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 1 FOLLOWER]: Refusing update from remote peer 0cc256c0971c4ff7ad7d7f97bd9a13b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:07.271736 4783 raft_consensus.cc:1261] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 1 FOLLOWER]: Refusing update from remote peer 0cc256c0971c4ff7ad7d7f97bd9a13b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:07.272568 4988 consensus_queue.cc:1035] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db193e62c2e64ef6a862cbe8845ac33c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:07.272838 4986 consensus_queue.cc:1035] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c9c67712504a0b93962069c3ca4264" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:07.287240 4991 mvcc.cc:205] Tried to move back new op lower bound from 7332864275536596992 to 7332864275429629952. Current Snapshot: MvccSnapshot[applied={T|T < 7332864275536596992}]
I20260924 11:41:07.294112 4992 mvcc.cc:205] Tried to move back new op lower bound from 7332864275536596992 to 7332864275429629952. Current Snapshot: MvccSnapshot[applied={T|T < 7332864275536596992 or (T in {7332864275536596992})}]
I20260924 11:41:07.301788 4507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:07.302047 4507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:07.304983 4507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5430774df5354bf8a82136c487f4d9d4: Sending DeleteTablet for 3 replicas of tablet 05d24fed29fa4ecdadde18bd6e9d3477
I20260924 11:41:07.306118 4763 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d24fed29fa4ecdadde18bd6e9d3477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34926
I20260924 11:41:07.306141 4895 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d24fed29fa4ecdadde18bd6e9d3477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46800
I20260924 11:41:07.306895 4631 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d24fed29fa4ecdadde18bd6e9d3477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43214
I20260924 11:41:07.307183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4569
I20260924 11:41:07.307344 5001 tablet_replica.cc:330] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: stopping tablet replica
I20260924 11:41:07.307727 5001 raft_consensus.cc:2229] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:07.308326 5001 raft_consensus.cc:2258] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:07.308969 5000 tablet_replica.cc:330] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: stopping tablet replica
I20260924 11:41:07.309290 5000 raft_consensus.cc:2229] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:07.309674 5000 raft_consensus.cc:2258] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:07.311874 5000 ts_tablet_manager.cc:1905] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:07.311889 5001 ts_tablet_manager.cc:1905] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:07.319458 5001 ts_tablet_manager.cc:1918] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:07.319598 5001 log.cc:1196] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/05d24fed29fa4ecdadde18bd6e9d3477
I20260924 11:41:07.320125 5001 ts_tablet_manager.cc:1939] T 05d24fed29fa4ecdadde18bd6e9d3477 P 0cc256c0971c4ff7ad7d7f97bd9a13b6: Deleting consensus metadata
I20260924 11:41:07.320516 5002 tablet_replica.cc:330] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: stopping tablet replica
I20260924 11:41:07.320878 5002 raft_consensus.cc:2229] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:07.321270 5002 raft_consensus.cc:2258] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:07.321597 4493 catalog_manager.cc:4841] TS 0cc256c0971c4ff7ad7d7f97bd9a13b6 (127.0.97.3:37485): tablet 05d24fed29fa4ecdadde18bd6e9d3477 (table test-table [id=422514afb9b5484db0a81e66b5f19d35]) successfully deleted
I20260924 11:41:07.321663 5000 ts_tablet_manager.cc:1918] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:07.321784 5000 log.cc:1196] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/05d24fed29fa4ecdadde18bd6e9d3477
I20260924 11:41:07.322253 5000 ts_tablet_manager.cc:1939] T 05d24fed29fa4ecdadde18bd6e9d3477 P db193e62c2e64ef6a862cbe8845ac33c: Deleting consensus metadata
I20260924 11:41:07.323701 4492 catalog_manager.cc:4841] TS db193e62c2e64ef6a862cbe8845ac33c (127.0.97.2:46713): tablet 05d24fed29fa4ecdadde18bd6e9d3477 (table test-table [id=422514afb9b5484db0a81e66b5f19d35]) successfully deleted
I20260924 11:41:07.323977 5002 ts_tablet_manager.cc:1905] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:41:07 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:07.331341 5002 ts_tablet_manager.cc:1918] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:07.331501 5002 log.cc:1196] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/05d24fed29fa4ecdadde18bd6e9d3477
I20260924 11:41:07.332031 5002 ts_tablet_manager.cc:1939] T 05d24fed29fa4ecdadde18bd6e9d3477 P 74c9c67712504a0b93962069c3ca4264: Deleting consensus metadata
I20260924 11:41:07.332789 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.309031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.0.97.1:43893)
0924 11:41:07.309182 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.309190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.316187 (+ 6997us) server_negotiation.cc:407] Connection header received
0924 11:41:07.316360 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.316365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.316468 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.316563 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:07.318092 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.319198 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.321573 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.322265 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.322388 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.326880 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:07.326889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:07.326899 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:07.326930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:07.330003 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:07.330589 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:07.330594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:07.330596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:07.330711 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:07.331064 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:07.331069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:07.331072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:07.331391 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:07.331725 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.332237 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.332434 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:41:07.333451 4691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:07.333504 4493 catalog_manager.cc:4841] TS 74c9c67712504a0b93962069c3ca4264 (127.0.97.1:43893): tablet 05d24fed29fa4ecdadde18bd6e9d3477 (table test-table [id=422514afb9b5484db0a81e66b5f19d35]) successfully deleted
I20260924 11:41:07.431766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4569
I20260924 11:41:07.448915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4701
Sep 24 11:41:07 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:41:07.462913 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.449616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.97.2:46713)
0924 11:41:07.449810 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.449818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.449838 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:41:07.450052 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.450059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.450157 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.450255 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:07.451397 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.452045 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.453310 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.453704 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.453838 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.457864 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:07.457876 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:07.457890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:07.457924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:07.460435 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:07.461016 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:07.461023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:07.461026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:07.461141 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:07.461497 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:07.461503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:07.461507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:07.461835 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:07.462161 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.462396 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.462593 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:41:07.463595 4823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:07.547495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4701
I20260924 11:41:07.564908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4833
Sep 24 11:41:07 dist-test-slave-qhrh krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:41:07.581303 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:07.565694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.0.97.3:37485)
0924 11:41:07.565893 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:07.565901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:07.565982 (+ 81us) server_negotiation.cc:407] Connection header received
0924 11:41:07.566199 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:07.566203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:07.566264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:07.566398 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:07.567633 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.568594 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.569876 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:07.570302 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:07.570431 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:07.574646 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:07.574659 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:07.574678 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:07.574714 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:07.578650 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:07.579201 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:07.579210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:07.579213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:07.579380 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:07.579726 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:07.579731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:07.579735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:07.580205 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:07.580506 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:07.580656 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 11:41:07.580837 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 11:41:07.582005 4955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:07.692704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4833
I20260924 11:41:07.706985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 4477
I20260924 11:41:07.708001 4538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:07.805548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 4477
2026-09-24T11:41:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3867 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T11:41:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:07Z Disabled control of system clock
I20260924 11:41:07.867002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36275
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 11:41:08.157298 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:08.157546 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:08.157655 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:08.164283 5011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:08.164369 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:08.164405 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:08.164465 5011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:08.164505 5011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:08.172979 5011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:08.173995 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:08.175577 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:08.186189 5016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:08.186229 5017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:08.186270 5019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:08.186910 5011 server_base.cc:1034] running on GCE node
I20260924 11:41:08.187615 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:08.188827 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:08.190037 5011 hybrid_clock.cc:648] HybridClock initialized: now 1790250068190006 us; error 40 us; skew 500 ppm
I20260924 11:41:08.190380 5011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:08.192490 5011 webserver.cc:460] Webserver started at http://127.0.97.62:37975/ using document root <none> and password file <none>
I20260924 11:41:08.193074 5011 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:08.193190 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:08.193487 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:08.196120 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "874f0b8540d54464a980acc6b197b816"
format_stamp: "Formatted at 2026-09-24 11:41:08 on dist-test-slave-qhrh"
server_key: "a0a97005bb6a487a176dfe5b5aadb0a1"
server_key_iv: "c7766b4edcfaacb9bdf06328176aecb8"
server_key_version: "encryptionkey@0"
I20260924 11:41:08.196921 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "874f0b8540d54464a980acc6b197b816"
format_stamp: "Formatted at 2026-09-24 11:41:08 on dist-test-slave-qhrh"
server_key: "a0a97005bb6a487a176dfe5b5aadb0a1"
server_key_iv: "c7766b4edcfaacb9bdf06328176aecb8"
server_key_version: "encryptionkey@0"
I20260924 11:41:08.201327 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:41:08.204231 5025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:08.205358 5011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:08.205528 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "874f0b8540d54464a980acc6b197b816"
format_stamp: "Formatted at 2026-09-24 11:41:08 on dist-test-slave-qhrh"
server_key: "a0a97005bb6a487a176dfe5b5aadb0a1"
server_key_iv: "c7766b4edcfaacb9bdf06328176aecb8"
server_key_version: "encryptionkey@0"
I20260924 11:41:08.205724 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:08.210811 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:08.211481 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:08.211736 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:08.227733 5011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36275
I20260924 11:41:08.227769 5076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36275 every 8 connection(s)
I20260924 11:41:08.229000 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:08.230926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5011
I20260924 11:41:08.231128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:08.231694 388 external_mini_cluster.cc:1428] Setting key 8a835a2f914062503d47d47170879a8b
I20260924 11:41:08.232441 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:08.239768 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:08.233635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.0.97.62:36275)
0924 11:41:08.234480 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:08.234490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:08.234526 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:41:08.236602 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:08.236624 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:08.236991 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:08.237312 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:08.238326 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:08.238331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:08.238345 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:08.238360 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:08.238417 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:08.238435 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:08.238592 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:08.238706 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:41:08.238978 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":384,"threads_started":1}
I20260924 11:41:08.244061 5077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816: Bootstrap starting.
I20260924 11:41:08.246687 5077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:08.247570 5077 log.cc:824] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:08.249689 5077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816: No bootstrap required, opened a new log
I20260924 11:41:08.256045 5077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874f0b8540d54464a980acc6b197b816" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36275 } }
I20260924 11:41:08.256418 5077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:08.256526 5077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874f0b8540d54464a980acc6b197b816, State: Initialized, Role: FOLLOWER
I20260924 11:41:08.257056 5077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874f0b8540d54464a980acc6b197b816" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36275 } }
I20260924 11:41:08.257261 5077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:08.257401 5077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:08.257555 5077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:08.259322 5077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874f0b8540d54464a980acc6b197b816" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36275 } }
I20260924 11:41:08.259758 5077 leader_election.cc:302] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874f0b8540d54464a980acc6b197b816; no voters:
I20260924 11:41:08.260299 5077 leader_election.cc:288] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:08.260385 5082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:08.261086 5082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [term 1 LEADER]: Becoming Leader. State: Replica: 874f0b8540d54464a980acc6b197b816, State: Running, Role: LEADER
I20260924 11:41:08.261581 5082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874f0b8540d54464a980acc6b197b816" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36275 } }
I20260924 11:41:08.262089 5077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:08.263880 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874f0b8540d54464a980acc6b197b816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874f0b8540d54464a980acc6b197b816" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36275 } } }
I20260924 11:41:08.264209 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:08.264887 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874f0b8540d54464a980acc6b197b816. Latest consensus state: current_term: 1 leader_uuid: "874f0b8540d54464a980acc6b197b816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874f0b8540d54464a980acc6b197b816" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36275 } } }
I20260924 11:41:08.265172 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:08.267553 5088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:08.271493 5088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:08.278474 5088 catalog_manager.cc:1347] Generated new cluster ID: 4c3e778cd3174ba88fd1a5fc522fc81c
I20260924 11:41:08.278574 5088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:08.302603 5088 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:08.302949 5088 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:08.309079 5088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816: Generated new TSK 0
I20260924 11:41:08.309758 5088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:41:08.322012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36275
--builtin_ntp_servers=127.0.97.20:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 11:41:08.631351 5101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:08.631609 5101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:08.631775 5101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:08.638808 5101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:08.638899 5101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:08.639118 5101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:08.647841 5101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:08.649008 5101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:08.650719 5101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:08.663947 5106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:08.663880 5107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:08.663924 5109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:08.665194 5101 server_base.cc:1034] running on GCE node
I20260924 11:41:08.665730 5101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:08.666651 5101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:08.667886 5101 hybrid_clock.cc:648] HybridClock initialized: now 1790250068667833 us; error 36 us; skew 500 ppm
I20260924 11:41:08.668241 5101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:08.670342 5101 webserver.cc:460] Webserver started at http://127.0.97.1:42141/ using document root <none> and password file <none>
I20260924 11:41:08.670989 5101 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:08.671111 5101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:08.671425 5101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:08.674080 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "23261cb17d4442b5bee4481966427cd5"
format_stamp: "Formatted at 2026-09-24 11:41:08 on dist-test-slave-qhrh"
server_key: "0afd115de8b58aefc48fe2dee7fc8617"
server_key_iv: "f4f5a73785ecbe31eb9d5e3b0b2c8cd1"
server_key_version: "encryptionkey@0"
I20260924 11:41:08.674821 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "23261cb17d4442b5bee4481966427cd5"
format_stamp: "Formatted at 2026-09-24 11:41:08 on dist-test-slave-qhrh"
server_key: "0afd115de8b58aefc48fe2dee7fc8617"
server_key_iv: "f4f5a73785ecbe31eb9d5e3b0b2c8cd1"
server_key_version: "encryptionkey@0"
I20260924 11:41:08.679363 5101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 11:41:08.682480 5115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:08.683647 5101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:08.683815 5101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "23261cb17d4442b5bee4481966427cd5"
format_stamp: "Formatted at 2026-09-24 11:41:08 on dist-test-slave-qhrh"
server_key: "0afd115de8b58aefc48fe2dee7fc8617"
server_key_iv: "f4f5a73785ecbe31eb9d5e3b0b2c8cd1"
server_key_version: "encryptionkey@0"
I20260924 11:41:08.684031 5101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:08.689193 5101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:08.689893 5101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:08.690147 5101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:08.690994 5101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:08.692250 5101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:08.692371 5101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:08.692504 5101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:08.692590 5101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:08.723703 5101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46139
I20260924 11:41:08.723801 5227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46139 every 8 connection(s)
I20260924 11:41:08.725023 5101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:08.728315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5101
I20260924 11:41:08.728549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:08.729372 388 external_mini_cluster.cc:1428] Setting key 20d73b77c29fa0c5eea5c8f4cdd6ac3d
I20260924 11:41:08.731900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36275
--builtin_ntp_servers=127.0.97.20:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 11:41:08.733745 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:08.727539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41803 (local address 127.0.97.62:36275)
0924 11:41:08.727744 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:08.727749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:08.729363 (+ 1614us) server_negotiation.cc:407] Connection header received
0924 11:41:08.731336 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:08.731342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:08.731411 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:08.731537 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:08.732966 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:08.732973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:08.732978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:08.733005 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:08.733038 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:08.733051 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:08.733178 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:08.733391 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 11:41:08.733457 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:41:08.734640 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:08.728128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36275 (local address 127.0.97.1:41803)
0924 11:41:08.729192 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:08.729234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:08.731075 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:08.731760 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:08.731768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:08.731858 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:08.732675 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:08.732678 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:08.732712 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:08.732796 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:08.733153 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:08.733163 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:08.733172 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 11:41:08.733473 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 11:41:08.733713 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":358,"threads_started":1}
I20260924 11:41:08.736117 5228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36275
I20260924 11:41:08.736490 5228 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:08.737071 5228 heartbeater.cc:511] Master 127.0.97.62:36275 requested a full tablet report, sending...
I20260924 11:41:08.739202 5042 ts_manager.cc:194] Registered new tserver with Master: 23261cb17d4442b5bee4481966427cd5 (127.0.97.1:46139)
W20260924 11:41:09.045063 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:09.045312 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:09.045480 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:09.052526 5231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:09.052605 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:09.052788 5231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:09.061528 5231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:09.062633 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:09.064318 5231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:09.076665 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:09.076725 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:09.076733 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:09.077500 5231 server_base.cc:1034] running on GCE node
I20260924 11:41:09.078132 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:09.078989 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:09.080245 5231 hybrid_clock.cc:648] HybridClock initialized: now 1790250069080182 us; error 49 us; skew 500 ppm
I20260924 11:41:09.080557 5231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:09.082638 5231 webserver.cc:460] Webserver started at http://127.0.97.2:33767/ using document root <none> and password file <none>
I20260924 11:41:09.083297 5231 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:09.083406 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:09.083700 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:09.086375 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "135f57aca04f40aba43766c680fd47e4"
format_stamp: "Formatted at 2026-09-24 11:41:09 on dist-test-slave-qhrh"
server_key: "478aaa5578350527173af2e21455632d"
server_key_iv: "f3e1ed1d0a09d71d667a9684304944f7"
server_key_version: "encryptionkey@0"
I20260924 11:41:09.087200 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "135f57aca04f40aba43766c680fd47e4"
format_stamp: "Formatted at 2026-09-24 11:41:09 on dist-test-slave-qhrh"
server_key: "478aaa5578350527173af2e21455632d"
server_key_iv: "f3e1ed1d0a09d71d667a9684304944f7"
server_key_version: "encryptionkey@0"
I20260924 11:41:09.091760 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 11:41:09.094926 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:09.096231 5231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:41:09.096393 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "135f57aca04f40aba43766c680fd47e4"
format_stamp: "Formatted at 2026-09-24 11:41:09 on dist-test-slave-qhrh"
server_key: "478aaa5578350527173af2e21455632d"
server_key_iv: "f3e1ed1d0a09d71d667a9684304944f7"
server_key_version: "encryptionkey@0"
I20260924 11:41:09.096616 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:09.101783 5231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:09.102511 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:09.102766 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:09.103579 5231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:09.104856 5231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:09.104983 5231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:09.105114 5231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:09.105201 5231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:09.136397 5231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44281
I20260924 11:41:09.136476 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44281 every 8 connection(s)
I20260924 11:41:09.137640 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:09.139959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5231
I20260924 11:41:09.140193 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:09.140614 388 external_mini_cluster.cc:1428] Setting key 6da0807f521f2f0d3d10d8c83e7f4907
I20260924 11:41:09.143525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36275
--builtin_ntp_servers=127.0.97.20:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 11:41:09.146498 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.139990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56797 (local address 127.0.97.62:36275)
0924 11:41:09.140207 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.140213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.141719 (+ 1506us) server_negotiation.cc:407] Connection header received
0924 11:41:09.143797 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.143805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.143866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.143933 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.145616 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.145621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.145624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.145687 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.145730 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.145749 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.145894 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.146099 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.146160 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:41:09.147535 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.140546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36275 (local address 127.0.97.2:56797)
0924 11:41:09.141563 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.141609 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.143486 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.144427 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.144441 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.144560 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.145251 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.145257 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.145322 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.145435 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.145875 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.145887 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.145905 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.146156 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.146394 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":336,"threads_started":1}
I20260924 11:41:09.148818 5358 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36275
I20260924 11:41:09.149091 5358 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:09.149631 5358 heartbeater.cc:511] Master 127.0.97.62:36275 requested a full tablet report, sending...
I20260924 11:41:09.150986 5042 ts_manager.cc:194] Registered new tserver with Master: 135f57aca04f40aba43766c680fd47e4 (127.0.97.2:44281)
W20260924 11:41:09.461153 5361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:09.461400 5361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:09.461490 5361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:09.469110 5361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:09.469203 5361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:09.469316 5361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:09.478677 5361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:09.479768 5361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:09.481402 5361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:09.493566 5369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:09.493651 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:09.494026 5366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:09.494386 5361 server_base.cc:1034] running on GCE node
I20260924 11:41:09.494846 5361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:09.495623 5361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:09.496807 5361 hybrid_clock.cc:648] HybridClock initialized: now 1790250069496775 us; error 32 us; skew 500 ppm
I20260924 11:41:09.497071 5361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:09.498958 5361 webserver.cc:460] Webserver started at http://127.0.97.3:38963/ using document root <none> and password file <none>
I20260924 11:41:09.499666 5361 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:09.499744 5361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:09.500031 5361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:09.502592 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "5090d966edd047e58e6ae31c620b0122"
format_stamp: "Formatted at 2026-09-24 11:41:09 on dist-test-slave-qhrh"
server_key: "3592b6dc8be3b6bc555f9d1d07059c4c"
server_key_iv: "e287c205841d444c18b2b60002f820e5"
server_key_version: "encryptionkey@0"
I20260924 11:41:09.503288 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5090d966edd047e58e6ae31c620b0122"
format_stamp: "Formatted at 2026-09-24 11:41:09 on dist-test-slave-qhrh"
server_key: "3592b6dc8be3b6bc555f9d1d07059c4c"
server_key_iv: "e287c205841d444c18b2b60002f820e5"
server_key_version: "encryptionkey@0"
I20260924 11:41:09.507825 5361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260924 11:41:09.510989 5375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:09.512264 5361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:41:09.512404 5361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "5090d966edd047e58e6ae31c620b0122"
format_stamp: "Formatted at 2026-09-24 11:41:09 on dist-test-slave-qhrh"
server_key: "3592b6dc8be3b6bc555f9d1d07059c4c"
server_key_iv: "e287c205841d444c18b2b60002f820e5"
server_key_version: "encryptionkey@0"
I20260924 11:41:09.512634 5361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:09.517449 5361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:09.518167 5361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:09.518416 5361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:09.519245 5361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:09.520509 5361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:09.520622 5361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:09.520745 5361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:09.520838 5361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:09.550331 5361 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45507
I20260924 11:41:09.550397 5487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45507 every 8 connection(s)
I20260924 11:41:09.551627 5361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:09.558708 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.553916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52251 (local address 127.0.97.62:36275)
0924 11:41:09.554086 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.554090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.555263 (+ 1173us) server_negotiation.cc:407] Connection header received
0924 11:41:09.556887 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.556892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.556943 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.557026 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.558183 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.558187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.558190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.558202 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.558227 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.558237 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.558327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.558479 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.558524 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:41:09.559511 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.554304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36275 (local address 127.0.97.3:52251)
0924 11:41:09.555114 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.555146 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.556650 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.557200 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.557207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.557284 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.557941 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.557944 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.557977 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.558049 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.558315 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.558321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.558326 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.558519 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.558679 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":263,"threads_started":1}
I20260924 11:41:09.560751 5488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36275
I20260924 11:41:09.561003 5488 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:09.561457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5361
I20260924 11:41:09.561537 5488 heartbeater.cc:511] Master 127.0.97.62:36275 requested a full tablet report, sending...
I20260924 11:41:09.561651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:09.562168 388 external_mini_cluster.cc:1428] Setting key 1fb89cf6a1c99c967f75b7372d2fb666
I20260924 11:41:09.563059 5042 ts_manager.cc:194] Registered new tserver with Master: 5090d966edd047e58e6ae31c620b0122 (127.0.97.3:45507)
I20260924 11:41:09.565225 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:41:09.571744 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.569346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.0.97.62:36275)
0924 11:41:09.569481 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.569485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.569970 (+ 485us) server_negotiation.cc:407] Connection header received
0924 11:41:09.570158 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.570162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.570215 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.570305 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.571150 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.571155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.571159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.571175 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.571209 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.571220 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.571345 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.571439 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.571490 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:41:09.574905 5042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:09.580972 5042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:09.597800 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.590925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44281 (local address 127.0.0.1:37222)
0924 11:41:09.591763 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.591811 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.591966 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.593168 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.593176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.593260 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.594125 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.594131 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.594165 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.594246 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.596918 (+ 2672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.596932 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.596943 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.597115 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.597213 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":425,"threads_started":1}
I20260924 11:41:09.600446 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.591178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.97.2:44281)
0924 11:41:09.592550 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.592557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.592588 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:41:09.592650 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.592657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.592865 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.593718 (+ 853us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.596594 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.596604 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.596621 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.596667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.596738 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.596760 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.598664 (+ 1904us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.599788 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.599947 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":1077,"threads_started":1}
I20260924 11:41:09.604396 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.593519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53496)
0924 11:41:09.598072 (+ 4553us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.598093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.598183 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.601408 (+ 3225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.601412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.601453 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.602014 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.602017 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.602034 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.602073 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.604014 (+ 1941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.604019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.604023 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.604094 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.604135 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4430,"threads_started":1}
I20260924 11:41:09.604310 5293 tablet_service.cc:1467] Processing CreateTablet for tablet dfbd73c412ab4bbba70aba06274439b8 (DEFAULT_TABLE table=test-table [id=d91db60fc0b64385954628ce6c129ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:09.605044 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.593911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.0.97.1:46139)
0924 11:41:09.600953 (+ 7042us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.600961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.600987 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:41:09.601043 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.601049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.601238 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.601700 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.603753 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.603761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.603772 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.603789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.603842 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.603861 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.604603 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.604676 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.604753 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6890,"threads_started":1}
I20260924 11:41:09.605749 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbd73c412ab4bbba70aba06274439b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:09.608068 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.592272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.0.97.3:45507)
0924 11:41:09.605279 (+ 13007us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.605286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.605316 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:41:09.605399 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.605403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.605529 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.605676 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.607249 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.607257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.607270 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.607285 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.607331 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.607348 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.607551 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.607657 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.607735 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12867,"thread_start_us":12705,"threads_started":1}
I20260924 11:41:09.608987 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.591225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45507 (local address 127.0.0.1:36524)
0924 11:41:09.593316 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.593336 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.593435 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.605673 (+ 12238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.605677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.605699 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.606360 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.606365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.606383 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.606437 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.607477 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.607484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.607507 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.608539 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.608584 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1969,"thread_start_us":1801,"threads_started":1}
I20260924 11:41:09.608762 5163 tablet_service.cc:1467] Processing CreateTablet for tablet dfbd73c412ab4bbba70aba06274439b8 (DEFAULT_TABLE table=test-table [id=d91db60fc0b64385954628ce6c129ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:09.609993 5163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbd73c412ab4bbba70aba06274439b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:09.612408 5423 tablet_service.cc:1467] Processing CreateTablet for tablet dfbd73c412ab4bbba70aba06274439b8 (DEFAULT_TABLE table=test-table [id=d91db60fc0b64385954628ce6c129ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:09.613659 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbd73c412ab4bbba70aba06274439b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:09.616950 5506 tablet_bootstrap.cc:492] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Bootstrap starting.
I20260924 11:41:09.620627 5506 tablet_bootstrap.cc:654] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:09.621795 5506 log.cc:824] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:09.625510 5508 tablet_bootstrap.cc:492] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Bootstrap starting.
I20260924 11:41:09.626543 5506 tablet_bootstrap.cc:492] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: No bootstrap required, opened a new log
I20260924 11:41:09.626808 5506 ts_tablet_manager.cc:1397] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260924 11:41:09.627410 5509 tablet_bootstrap.cc:492] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Bootstrap starting.
I20260924 11:41:09.628934 5508 tablet_bootstrap.cc:654] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:09.630203 5508 log.cc:824] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:09.630442 5509 tablet_bootstrap.cc:654] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:09.631609 5509 log.cc:824] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:09.633001 5508 tablet_bootstrap.cc:492] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: No bootstrap required, opened a new log
I20260924 11:41:09.633244 5508 ts_tablet_manager.cc:1397] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:41:09.633980 5509 tablet_bootstrap.cc:492] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: No bootstrap required, opened a new log
I20260924 11:41:09.634222 5509 ts_tablet_manager.cc:1397] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 11:41:09.636660 5506 raft_consensus.cc:348] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.637089 5506 raft_consensus.cc:374] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:09.637184 5506 raft_consensus.cc:729] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135f57aca04f40aba43766c680fd47e4, State: Initialized, Role: FOLLOWER
I20260924 11:41:09.637775 5506 consensus_queue.cc:260] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.638854 5358 heartbeater.cc:503] Master 127.0.97.62:36275 was elected leader, sending a full tablet report...
I20260924 11:41:09.639375 5506 ts_tablet_manager.cc:1428] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Time spent starting tablet: real 0.012s user 0.012s sys 0.002s
W20260924 11:41:09.640312 5359 tablet.cc:2369] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:09.642248 5509 raft_consensus.cc:348] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.642597 5509 raft_consensus.cc:374] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:09.642714 5509 raft_consensus.cc:729] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5090d966edd047e58e6ae31c620b0122, State: Initialized, Role: FOLLOWER
I20260924 11:41:09.643023 5508 raft_consensus.cc:348] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.643412 5508 raft_consensus.cc:374] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:09.643509 5508 raft_consensus.cc:729] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23261cb17d4442b5bee4481966427cd5, State: Initialized, Role: FOLLOWER
I20260924 11:41:09.643389 5509 consensus_queue.cc:260] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.644017 5508 consensus_queue.cc:260] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.644585 5488 heartbeater.cc:503] Master 127.0.97.62:36275 was elected leader, sending a full tablet report...
I20260924 11:41:09.644994 5509 ts_tablet_manager.cc:1428] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260924 11:41:09.649025 5228 heartbeater.cc:503] Master 127.0.97.62:36275 was elected leader, sending a full tablet report...
I20260924 11:41:09.649191 5508 ts_tablet_manager.cc:1428] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260924 11:41:09.727296 5229 tablet.cc:2369] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:09.774331 5512 raft_consensus.cc:482] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:09.774561 5512 raft_consensus.cc:504] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.775890 5512 leader_election.cc:288] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5090d966edd047e58e6ae31c620b0122 (127.0.97.3:45507), 23261cb17d4442b5bee4481966427cd5 (127.0.97.1:46139)
I20260924 11:41:09.778949 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.776327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43347 (local address 127.0.97.3:45507)
0924 11:41:09.776585 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.776593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.776898 (+ 305us) server_negotiation.cc:407] Connection header received
0924 11:41:09.777053 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.777061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.777136 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.777239 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.778330 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.778337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.778341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.778359 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.778395 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.778412 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.778525 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.778622 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.778663 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:41:09.779075 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.776222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45507 (local address 127.0.97.2:43347)
0924 11:41:09.776762 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.776776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.776923 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.777296 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.777300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.777322 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.778069 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.778077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.778097 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.778148 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.778501 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.778506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.778510 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.778618 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.778662 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":324,"threads_started":1}
I20260924 11:41:09.779412 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.777007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43295 (local address 127.0.97.1:46139)
0924 11:41:09.777191 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.777197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.777542 (+ 345us) server_negotiation.cc:407] Connection header received
0924 11:41:09.777689 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.777696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.777745 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.777836 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.778767 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.778772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.778776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.778811 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.778842 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.778855 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.778964 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.779128 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.779184 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:41:09.780016 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.776910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.97.2:43295)
0924 11:41:09.777443 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.777460 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:09.777567 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:09.777892 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:09.777896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:09.777916 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:09.778600 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:09.778602 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:09.778614 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:09.778647 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:09.778961 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:09.778969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:09.778975 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:41:09.779656 (+ 681us) client_negotiation.cc:240] Negotiation successful
0924 11:41:09.779700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":250,"threads_started":1}
I20260924 11:41:09.780028 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd73c412ab4bbba70aba06274439b8" candidate_uuid: "135f57aca04f40aba43766c680fd47e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5090d966edd047e58e6ae31c620b0122" is_pre_election: true
I20260924 11:41:09.780543 5443 raft_consensus.cc:2454] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135f57aca04f40aba43766c680fd47e4 in term 0.
I20260924 11:41:09.780825 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd73c412ab4bbba70aba06274439b8" candidate_uuid: "135f57aca04f40aba43766c680fd47e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23261cb17d4442b5bee4481966427cd5" is_pre_election: true
I20260924 11:41:09.781164 5246 leader_election.cc:302] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135f57aca04f40aba43766c680fd47e4, 5090d966edd047e58e6ae31c620b0122; no voters:
I20260924 11:41:09.781309 5183 raft_consensus.cc:2454] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135f57aca04f40aba43766c680fd47e4 in term 0.
I20260924 11:41:09.781522 5512 raft_consensus.cc:2781] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:09.781683 5512 raft_consensus.cc:482] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:09.781792 5512 raft_consensus.cc:3037] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:09.783736 5512 raft_consensus.cc:504] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.784307 5512 leader_election.cc:288] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [CANDIDATE]: Term 1 election: Requested vote from peers 5090d966edd047e58e6ae31c620b0122 (127.0.97.3:45507), 23261cb17d4442b5bee4481966427cd5 (127.0.97.1:46139)
I20260924 11:41:09.784643 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd73c412ab4bbba70aba06274439b8" candidate_uuid: "135f57aca04f40aba43766c680fd47e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5090d966edd047e58e6ae31c620b0122"
I20260924 11:41:09.784718 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd73c412ab4bbba70aba06274439b8" candidate_uuid: "135f57aca04f40aba43766c680fd47e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23261cb17d4442b5bee4481966427cd5"
I20260924 11:41:09.784821 5443 raft_consensus.cc:3037] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:09.784862 5183 raft_consensus.cc:3037] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:09.787936 5183 raft_consensus.cc:2454] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135f57aca04f40aba43766c680fd47e4 in term 1.
I20260924 11:41:09.787936 5443 raft_consensus.cc:2454] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135f57aca04f40aba43766c680fd47e4 in term 1.
I20260924 11:41:09.788252 5246 leader_election.cc:302] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135f57aca04f40aba43766c680fd47e4, 5090d966edd047e58e6ae31c620b0122; no voters:
I20260924 11:41:09.788479 5512 raft_consensus.cc:2781] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:09.788877 5512 raft_consensus.cc:686] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 1 LEADER]: Becoming Leader. State: Replica: 135f57aca04f40aba43766c680fd47e4, State: Running, Role: LEADER
I20260924 11:41:09.789453 5512 consensus_queue.cc:237] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } }
I20260924 11:41:09.793496 5041 catalog_manager.cc:5495] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 135f57aca04f40aba43766c680fd47e4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "135f57aca04f40aba43766c680fd47e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135f57aca04f40aba43766c680fd47e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } health_report { overall_health: UNKNOWN } } }
W20260924 11:41:09.803203 5489 tablet.cc:2369] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:09.833323 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.831652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.97.2:44281)
0924 11:41:09.831799 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.831803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.831818 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:41:09.831883 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.831886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.831948 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.832021 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.832814 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.832817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.832820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.832830 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.832852 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.832860 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.832961 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.833041 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.833078 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:41:09.840485 5183 raft_consensus.cc:1261] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 1 FOLLOWER]: Refusing update from remote peer 135f57aca04f40aba43766c680fd47e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:09.840523 5443 raft_consensus.cc:1261] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 1 FOLLOWER]: Refusing update from remote peer 135f57aca04f40aba43766c680fd47e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:09.841317 5517 consensus_queue.cc:1035] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5090d966edd047e58e6ae31c620b0122" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:09.841634 5512 consensus_queue.cc:1035] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23261cb17d4442b5bee4481966427cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:09.856895 5522 mvcc.cc:205] Tried to move back new op lower bound from 7332864286057582592 to 7332864285862014976. Current Snapshot: MvccSnapshot[applied={T|T < 7332864286057582592}]
I20260924 11:41:09.867694 5042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:09.867970 5042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:09.870772 5042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874f0b8540d54464a980acc6b197b816: Sending DeleteTablet for 3 replicas of tablet dfbd73c412ab4bbba70aba06274439b8
I20260924 11:41:09.871701 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbd73c412ab4bbba70aba06274439b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:09 UTC) from {username='slave'} at 127.0.0.1:37222
I20260924 11:41:09.871781 5163 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbd73c412ab4bbba70aba06274439b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:09 UTC) from {username='slave'} at 127.0.0.1:53496
I20260924 11:41:09.872334 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbd73c412ab4bbba70aba06274439b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:09 UTC) from {username='slave'} at 127.0.0.1:36524
I20260924 11:41:09.873123 5533 tablet_replica.cc:330] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: stopping tablet replica
I20260924 11:41:09.873275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5101
I20260924 11:41:09.873453 5533 raft_consensus.cc:2229] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:09.873827 5533 raft_consensus.cc:2258] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:09.876001 5533 ts_tablet_manager.cc:1905] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:09.880539 5531 tablet_replica.cc:330] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: stopping tablet replica
I20260924 11:41:09.880872 5531 raft_consensus.cc:2229] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:09.881363 5531 raft_consensus.cc:2258] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:09.881986 5532 tablet_replica.cc:330] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: stopping tablet replica
I20260924 11:41:09.882270 5532 raft_consensus.cc:2229] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:09.882625 5532 raft_consensus.cc:2258] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:09.882809 5533 ts_tablet_manager.cc:1918] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:09.882936 5533 log.cc:1196] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/dfbd73c412ab4bbba70aba06274439b8
I20260924 11:41:09.883427 5533 ts_tablet_manager.cc:1939] T dfbd73c412ab4bbba70aba06274439b8 P 5090d966edd047e58e6ae31c620b0122: Deleting consensus metadata
I20260924 11:41:09.884455 5532 ts_tablet_manager.cc:1905] T dfbd73c412ab4bbba70aba06274439b8 P 23261cb17d4442b5bee4481966427cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:09.885010 5026 catalog_manager.cc:4841] TS 5090d966edd047e58e6ae31c620b0122 (127.0.97.3:45507): tablet dfbd73c412ab4bbba70aba06274439b8 (table test-table [id=d91db60fc0b64385954628ce6c129ad1]) successfully deleted
I20260924 11:41:09.885125 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.874626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.0.97.1:46139)
0924 11:41:09.874801 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.874807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.880622 (+ 5815us) server_negotiation.cc:407] Connection header received
0924 11:41:09.881118 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.881126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.881235 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.881314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.884301 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.884307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.884312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.884331 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.884367 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.884381 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.884497 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.884799 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.884855 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:41:09.886152 5531 ts_tablet_manager.cc:1905] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:09.886216 5223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:09.892935 5531 ts_tablet_manager.cc:1918] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:09.893070 5531 log.cc:1196] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/dfbd73c412ab4bbba70aba06274439b8
I20260924 11:41:09.893559 5531 ts_tablet_manager.cc:1939] T dfbd73c412ab4bbba70aba06274439b8 P 135f57aca04f40aba43766c680fd47e4: Deleting consensus metadata
I20260924 11:41:09.895104 5027 catalog_manager.cc:4841] TS 135f57aca04f40aba43766c680fd47e4 (127.0.97.2:44281): tablet dfbd73c412ab4bbba70aba06274439b8 (table test-table [id=d91db60fc0b64385954628ce6c129ad1]) successfully deleted
I20260924 11:41:09.971498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5101
W20260924 11:41:09.994426 5026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46139 (error 108)
I20260924 11:41:09.995162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5231
I20260924 11:41:09.996794 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.996245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53516)
0924 11:41:09.996483 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:09.996620 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 11:41:09.997088 5026 catalog_manager.cc:4584] TS 23261cb17d4442b5bee4481966427cd5 (127.0.97.1:46139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfbd73c412ab4bbba70aba06274439b8: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
I20260924 11:41:09.998478 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:09.996074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.0.97.2:44281)
0924 11:41:09.996290 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:09.996298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:09.996410 (+ 112us) server_negotiation.cc:407] Connection header received
0924 11:41:09.996595 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:09.996601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:09.996671 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:09.996792 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:09.997753 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:09.997758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:09.997761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:09.997776 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:09.997829 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:09.997842 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:09.997961 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:09.998084 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:41:09.998131 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 11:41:09.999197 5353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:10.050740 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.050241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53526)
0924 11:41:10.050478 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.050568 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 11:41:10.051692 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.051366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53536)
0924 11:41:10.051511 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.051590 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:41:10.105041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5231
I20260924 11:41:10.111970 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.111552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53548)
0924 11:41:10.111743 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.111816 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:41:10.112926 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.112609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53550)
0924 11:41:10.112755 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.112814 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:41:10.123292 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5361
I20260924 11:41:10.126001 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.123982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.97.3:45507)
0924 11:41:10.124182 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:10.124190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:10.124211 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:10.124409 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:10.124414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:10.124494 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:10.124599 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:10.125356 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:10.125362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:10.125366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:10.125386 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:10.125422 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:10.125434 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:10.125563 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:10.125656 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 11:41:10.125712 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:41:10.126591 5483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:10.193032 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.192611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53560)
0924 11:41:10.192794 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.192877 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:41:10.194046 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.193695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:53568)
0924 11:41:10.193875 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.193944 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 11:41:10.213425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5361
I20260924 11:41:10.230094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5011
I20260924 11:41:10.231269 5072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:10.333215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5011
2026-09-24T11:41:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2527 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 11:41:10.372908 388 mini_oidc.cc:152] Starting JWKS server
I20260924 11:41:10.375399 388 webserver.cc:460] Webserver started at https://127.0.0.1:36545/ using document root <none> and password file <none>
I20260924 11:41:10.375702 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:41:10.376358 388 webserver.cc:460] Webserver started at http://127.0.0.1:32823/ using document root <none> and password file <none>
2026-09-24T11:41:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:10Z Disabled control of system clock
I20260924 11:41:10.399158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42811
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/testchainca.cert with env {}
W20260924 11:41:10.714862 5545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:10.715310 5545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:10.715498 5545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:10.728111 5545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:10.728281 5545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:10.728336 5545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:10.728394 5545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:10.728435 5545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:41:10.728492 5545 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:41:10.728529 5545 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36545/jwks.json
I20260924 11:41:10.738063 5545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:10.739215 5545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:10.740525 5545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:10.751389 5550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:10.751554 5553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:10.751752 5551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:10.752434 5545 server_base.cc:1034] running on GCE node
I20260924 11:41:10.753152 5545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:10.754411 5545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:10.755599 5545 hybrid_clock.cc:648] HybridClock initialized: now 1790250070755561 us; error 29 us; skew 500 ppm
I20260924 11:41:10.755929 5545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:10.758011 5545 webserver.cc:460] Webserver started at http://127.0.97.62:37249/ using document root <none> and password file <none>
I20260924 11:41:10.758615 5545 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:10.758721 5545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:10.759021 5545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:10.761623 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "ad516ab2a1434ec39fcb6fb46e744d96"
format_stamp: "Formatted at 2026-09-24 11:41:10 on dist-test-slave-qhrh"
server_key: "3b317fc5ddf462198f85e107cbde773d"
server_key_iv: "b726c2d7781802d5448173940c270c82"
server_key_version: "encryptionkey@0"
I20260924 11:41:10.762355 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad516ab2a1434ec39fcb6fb46e744d96"
format_stamp: "Formatted at 2026-09-24 11:41:10 on dist-test-slave-qhrh"
server_key: "3b317fc5ddf462198f85e107cbde773d"
server_key_iv: "b726c2d7781802d5448173940c270c82"
server_key_version: "encryptionkey@0"
I20260924 11:41:10.767077 5545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:41:10.770169 5559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:10.771420 5545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:10.771595 5545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "ad516ab2a1434ec39fcb6fb46e744d96"
format_stamp: "Formatted at 2026-09-24 11:41:10 on dist-test-slave-qhrh"
server_key: "3b317fc5ddf462198f85e107cbde773d"
server_key_iv: "b726c2d7781802d5448173940c270c82"
server_key_version: "encryptionkey@0"
I20260924 11:41:10.771783 5545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:10.796131 5545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:10.796892 5545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:10.797142 5545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:10.814482 5545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42811
I20260924 11:41:10.814512 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42811 every 8 connection(s)
I20260924 11:41:10.815634 5545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:10.818974 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:10.824681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5545
I20260924 11:41:10.824872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:10.825495 388 external_mini_cluster.cc:1428] Setting key 111b55eff7de4833a5afcb2de1f45d17
I20260924 11:41:10.829034 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96: Bootstrap starting.
I20260924 11:41:10.831794 5611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:10.832794 5611 log.cc:824] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:10.835178 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96: No bootstrap required, opened a new log
I20260924 11:41:10.838245 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.827175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.0.97.62:42811)
0924 11:41:10.828051 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:10.828061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:10.828097 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:41:10.829787 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:10.829807 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:10.830225 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:10.830578 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:10.831116 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:10.832401 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:10.834115 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:10.834720 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:10.834936 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:10.836054 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:10.836061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:10.836073 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:10.836639 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:10.836713 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:10.836735 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:10.836877 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:10.837054 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 11:41:10.837326 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":322,"threads_started":1}
I20260924 11:41:10.841701 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:41:10.843389 5611 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42811 } }
I20260924 11:41:10.843704 5611 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:10.843801 5611 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad516ab2a1434ec39fcb6fb46e744d96, State: Initialized, Role: FOLLOWER
I20260924 11:41:10.844434 5611 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42811 } }
I20260924 11:41:10.844697 5611 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:10.844867 5611 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:10.845043 5611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:10.847039 5611 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42811 } }
I20260924 11:41:10.847506 5611 leader_election.cc:302] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad516ab2a1434ec39fcb6fb46e744d96; no voters:
I20260924 11:41:10.848085 5611 leader_election.cc:288] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:10.848212 5616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:10.848875 5616 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [term 1 LEADER]: Becoming Leader. State: Replica: ad516ab2a1434ec39fcb6fb46e744d96, State: Running, Role: LEADER
I20260924 11:41:10.849350 5616 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42811 } }
I20260924 11:41:10.850104 5611 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:10.852034 5617 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42811 } } }
I20260924 11:41:10.852387 5617 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:10.852768 5618 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad516ab2a1434ec39fcb6fb46e744d96. Latest consensus state: current_term: 1 leader_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad516ab2a1434ec39fcb6fb46e744d96" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42811 } } }
I20260924 11:41:10.853237 5618 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:10.859552 5624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:10.864135 5624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:10.872596 5624 catalog_manager.cc:1347] Generated new cluster ID: 8a5dad7a79cb40df8189a6444ccbd619
I20260924 11:41:10.872684 5624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:10.894873 5624 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:10.896039 5624 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:10.907022 5624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad516ab2a1434ec39fcb6fb46e744d96: Generated new TSK 0
I20260924 11:41:10.907912 5624 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 11:41:10.946135 5641 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 11:41:10.946271 5641 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:41:10.947294 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:10.926743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.0.97.62:42811)
0924 11:41:10.926923 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:10.926929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:10.927421 (+ 492us) server_negotiation.cc:407] Connection header received
0924 11:41:10.927666 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:10.927671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:10.927762 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:10.927899 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:41:10.928414 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:10.929497 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:10.930870 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:10.931569 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:10.931700 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:10.932173 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:41:10.946756 (+ 14583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 11:41:10.947035 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 11:41:10.947402 5614 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 11:41:10.948590 5640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:10.926664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42811 (local address 127.0.0.1:45068)
0924 11:41:10.927323 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:10.927341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:10.927480 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:10.927922 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:10.927925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:10.927934 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:41:10.928283 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:10.928292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:10.929637 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:10.929640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:10.930743 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:10.930753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:10.931693 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:10.931742 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:10.931858 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:10.931909 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:41:10.946975 (+ 15066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 11:41:10.947186 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":381,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 11:41:10.950271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5545
I20260924 11:41:10.951318 5606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:11.041695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5545
2026-09-24T11:41:11Z chronyd exiting
I20260924 11:41:11.090914 388 test_util.cc:175] -----------------------------------------------
I20260924 11:41:11.091024 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250027667721-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 11:41:11.094275 388 mini_oidc.cc:152] Starting JWKS server
I20260924 11:41:11.095906 388 webserver.cc:460] Webserver started at https://127.0.0.1:42207/ using document root <none> and password file <none>
I20260924 11:41:11.096099 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:41:11.096786 388 webserver.cc:460] Webserver started at http://127.0.0.1:39771/ using document root <none> and password file <none>
2026-09-24T11:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:11Z Disabled control of system clock
I20260924 11:41:11.119046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41445
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41445
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42207/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/testchainca.cert with env {}
W20260924 11:41:11.411868 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:11.412146 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:11.412326 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:11.418691 5648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:11.418805 5648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:11.418892 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:11.418963 5648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:11.419035 5648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:41:11.419112 5648 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:41:11.419178 5648 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42207/jwks.json
I20260924 11:41:11.428062 5648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42207/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:11.429344 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:11.430699 5648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:11.440752 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:11.441609 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:11.442553 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:11.442796 5648 server_base.cc:1034] running on GCE node
I20260924 11:41:11.443401 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:11.444558 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:11.445742 5648 hybrid_clock.cc:648] HybridClock initialized: now 1790250071445696 us; error 45 us; skew 500 ppm
I20260924 11:41:11.446044 5648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:11.448037 5648 webserver.cc:460] Webserver started at http://127.0.97.62:41203/ using document root <none> and password file <none>
I20260924 11:41:11.448632 5648 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:11.448774 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:11.449080 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:11.451505 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "df9d6fa09aa243239f0d74a44889c07a"
format_stamp: "Formatted at 2026-09-24 11:41:11 on dist-test-slave-qhrh"
server_key: "8b12cd2c7b211af0e9a1fc0a418fde0d"
server_key_iv: "15b40dd8f0fd4d39a959dee00d8e967c"
server_key_version: "encryptionkey@0"
I20260924 11:41:11.452275 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "df9d6fa09aa243239f0d74a44889c07a"
format_stamp: "Formatted at 2026-09-24 11:41:11 on dist-test-slave-qhrh"
server_key: "8b12cd2c7b211af0e9a1fc0a418fde0d"
server_key_iv: "15b40dd8f0fd4d39a959dee00d8e967c"
server_key_version: "encryptionkey@0"
I20260924 11:41:11.456775 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:41:11.459652 5662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:11.460848 5648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:11.461021 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "df9d6fa09aa243239f0d74a44889c07a"
format_stamp: "Formatted at 2026-09-24 11:41:11 on dist-test-slave-qhrh"
server_key: "8b12cd2c7b211af0e9a1fc0a418fde0d"
server_key_iv: "15b40dd8f0fd4d39a959dee00d8e967c"
server_key_version: "encryptionkey@0"
I20260924 11:41:11.461208 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:11.498396 5648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:11.499102 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:11.499348 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:11.516014 5648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41445
I20260924 11:41:11.516038 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41445 every 8 connection(s)
I20260924 11:41:11.517199 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:11.520447 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:11.523941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5648
I20260924 11:41:11.524164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:11.524950 388 external_mini_cluster.cc:1428] Setting key a138e706510b30dac38bd6206ba5f427
I20260924 11:41:11.531064 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a: Bootstrap starting.
I20260924 11:41:11.534267 5714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:11.535403 5714 log.cc:824] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:11.538313 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a: No bootstrap required, opened a new log
I20260924 11:41:11.538657 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:11.526661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.0.97.62:41445)
0924 11:41:11.527549 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:11.527558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:11.527598 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:41:11.529763 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:11.529782 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:11.530162 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:11.530459 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:11.531027 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:11.532389 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:11.534041 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:11.534823 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:11.534952 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:11.536448 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:11.536463 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:11.536480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:11.537068 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:11.537131 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:11.537150 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:11.537252 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:11.537481 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 11:41:11.537773 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":364,"threads_started":1}
I20260924 11:41:11.541956 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:41:11.546473 5714 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9d6fa09aa243239f0d74a44889c07a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41445 } }
I20260924 11:41:11.546757 5714 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:11.546824 5714 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9d6fa09aa243239f0d74a44889c07a, State: Initialized, Role: FOLLOWER
I20260924 11:41:11.547307 5714 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9d6fa09aa243239f0d74a44889c07a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41445 } }
I20260924 11:41:11.547488 5714 raft_consensus.cc:388] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:11.547575 5714 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:11.547691 5714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:11.549892 5714 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9d6fa09aa243239f0d74a44889c07a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41445 } }
I20260924 11:41:11.550283 5714 leader_election.cc:302] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df9d6fa09aa243239f0d74a44889c07a; no voters:
I20260924 11:41:11.550748 5714 leader_election.cc:288] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:11.550886 5719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:11.551513 5719 raft_consensus.cc:686] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [term 1 LEADER]: Becoming Leader. State: Replica: df9d6fa09aa243239f0d74a44889c07a, State: Running, Role: LEADER
I20260924 11:41:11.552013 5719 consensus_queue.cc:237] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9d6fa09aa243239f0d74a44889c07a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41445 } }
I20260924 11:41:11.552615 5714 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:11.554929 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df9d6fa09aa243239f0d74a44889c07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9d6fa09aa243239f0d74a44889c07a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41445 } } }
I20260924 11:41:11.555375 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:11.555608 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df9d6fa09aa243239f0d74a44889c07a. Latest consensus state: current_term: 1 leader_uuid: "df9d6fa09aa243239f0d74a44889c07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9d6fa09aa243239f0d74a44889c07a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41445 } } }
I20260924 11:41:11.555887 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:11.557546 5728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:11.563068 5728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:11.571509 5728 catalog_manager.cc:1347] Generated new cluster ID: 7cbf6f41d20b414aa63cb8b9d43c235a
I20260924 11:41:11.571622 5728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:11.580830 5728 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:11.581843 5728 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:11.589123 5728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df9d6fa09aa243239f0d74a44889c07a: Generated new TSK 0
I20260924 11:41:11.589794 5728 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 11:41:11.627522 5744 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 11:41:11.627628 5744 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:41:11.628557 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:11.609320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.97.62:41445)
0924 11:41:11.609518 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:11.609525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:11.610120 (+ 595us) server_negotiation.cc:407] Connection header received
0924 11:41:11.610382 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:11.610388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:11.610467 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:11.610577 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:41:11.611096 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:11.611779 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:11.613813 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:11.614273 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:11.614364 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:11.614891 (+ 527us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:41:11.628041 (+ 13150us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42207/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 11:41:11.628371 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42207/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 11:41:11.628679 5717 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42207/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 11:41:11.629153 5743 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:11.609292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41445 (local address 127.0.0.1:52178)
0924 11:41:11.610011 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:11.610029 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:11.610170 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:11.610619 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:11.610626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:11.610639 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:41:11.610935 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:11.610947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:11.611918 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:11.611924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:11.613656 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:11.613687 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:11.614387 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:11.614433 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:11.614627 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:11.614669 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:41:11.628287 (+ 13618us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42207/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 11:41:11.628508 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41445: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42207/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":422,"threads_started":1}
I20260924 11:41:11.630431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5648
I20260924 11:41:11.631311 5709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:11.717743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5648
2026-09-24T11:41:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (678 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 11:41:11.773305 388 mini_oidc.cc:152] Starting JWKS server
I20260924 11:41:11.775081 388 webserver.cc:460] Webserver started at https://127.0.0.1:36001/ using document root <none> and password file <none>
I20260924 11:41:11.775332 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:41:11.775954 388 webserver.cc:460] Webserver started at http://127.0.0.1:39109/ using document root <none> and password file <none>
2026-09-24T11:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:11Z Disabled control of system clock
I20260924 11:41:11.798388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39347
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39347
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 11:41:12.093263 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:12.093529 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:12.093716 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:12.100448 5751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:12.100557 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:12.100632 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:12.100714 5751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:12.100771 5751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:41:12.100871 5751 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:41:12.100950 5751 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36001/jwks.json
I20260924 11:41:12.109185 5751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:12.110301 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:12.111915 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:12.122747 5756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:12.122780 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:12.122833 5757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:12.123678 5751 server_base.cc:1034] running on GCE node
I20260924 11:41:12.124449 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:12.125643 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:12.126859 5751 hybrid_clock.cc:648] HybridClock initialized: now 1790250072126828 us; error 41 us; skew 500 ppm
I20260924 11:41:12.127202 5751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:12.143306 5751 webserver.cc:460] Webserver started at http://127.0.97.62:46695/ using document root <none> and password file <none>
I20260924 11:41:12.143945 5751 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:12.144073 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:12.144459 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:12.147146 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "806c154ac2964fd6a613728e3962a098"
format_stamp: "Formatted at 2026-09-24 11:41:12 on dist-test-slave-qhrh"
server_key: "68f9001a604caa3ed9bf6544547dac78"
server_key_iv: "f99fc3e761d27d5d493b68414dff765c"
server_key_version: "encryptionkey@0"
I20260924 11:41:12.147889 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "806c154ac2964fd6a613728e3962a098"
format_stamp: "Formatted at 2026-09-24 11:41:12 on dist-test-slave-qhrh"
server_key: "68f9001a604caa3ed9bf6544547dac78"
server_key_iv: "f99fc3e761d27d5d493b68414dff765c"
server_key_version: "encryptionkey@0"
I20260924 11:41:12.152561 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:41:12.155577 5765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:12.156802 5751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:12.156985 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "806c154ac2964fd6a613728e3962a098"
format_stamp: "Formatted at 2026-09-24 11:41:12 on dist-test-slave-qhrh"
server_key: "68f9001a604caa3ed9bf6544547dac78"
server_key_iv: "f99fc3e761d27d5d493b68414dff765c"
server_key_version: "encryptionkey@0"
I20260924 11:41:12.157225 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:12.183694 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:12.184408 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:12.184651 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:12.201015 5751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39347
I20260924 11:41:12.201033 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39347 every 8 connection(s)
I20260924 11:41:12.202219 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:12.202657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5751
I20260924 11:41:12.202862 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:12.203619 388 external_mini_cluster.cc:1428] Setting key 42d32a304a668014f3954f6e7e578652
I20260924 11:41:12.206317 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:12.215284 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:12.205340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.0.97.62:39347)
0924 11:41:12.206215 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:12.206225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:12.206262 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:41:12.207730 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:12.207751 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:12.208188 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:12.208542 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:12.209056 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.210161 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.211303 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.211886 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.212086 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:12.213261 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:12.213267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:12.213278 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:12.213770 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:12.213837 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:12.213857 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:12.213996 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:12.214244 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 11:41:12.214494 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":348,"threads_started":1}
I20260924 11:41:12.216042 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098: Bootstrap starting.
I20260924 11:41:12.218667 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:41:12.219157 5817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:12.220064 5817 log.cc:824] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:12.222621 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098: No bootstrap required, opened a new log
I20260924 11:41:12.229365 5817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c154ac2964fd6a613728e3962a098" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260924 11:41:12.229666 5817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:12.229770 5817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806c154ac2964fd6a613728e3962a098, State: Initialized, Role: FOLLOWER
I20260924 11:41:12.230257 5817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c154ac2964fd6a613728e3962a098" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260924 11:41:12.230460 5817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:12.230597 5817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:12.230746 5817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:12.232860 5817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c154ac2964fd6a613728e3962a098" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260924 11:41:12.233295 5817 leader_election.cc:302] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 806c154ac2964fd6a613728e3962a098; no voters:
I20260924 11:41:12.233771 5817 leader_election.cc:288] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:12.233918 5822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:12.234556 5822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [term 1 LEADER]: Becoming Leader. State: Replica: 806c154ac2964fd6a613728e3962a098, State: Running, Role: LEADER
I20260924 11:41:12.235009 5822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c154ac2964fd6a613728e3962a098" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260924 11:41:12.235502 5817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:12.238283 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "806c154ac2964fd6a613728e3962a098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c154ac2964fd6a613728e3962a098" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } } }
I20260924 11:41:12.238334 5824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806c154ac2964fd6a613728e3962a098. Latest consensus state: current_term: 1 leader_uuid: "806c154ac2964fd6a613728e3962a098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c154ac2964fd6a613728e3962a098" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } } }
I20260924 11:41:12.238670 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:12.238744 5824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:12.240649 5832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:12.246176 5832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:12.253881 5832 catalog_manager.cc:1347] Generated new cluster ID: 3d86e593faf448768e7f5d1158817bc6
I20260924 11:41:12.253974 5832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:12.273402 5832 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:12.274514 5832 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:12.286068 5832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 806c154ac2964fd6a613728e3962a098: Generated new TSK 0
I20260924 11:41:12.286720 5832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:41:12.312808 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:12.304935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.0.97.62:39347)
0924 11:41:12.305129 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:12.305135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:12.305537 (+ 402us) server_negotiation.cc:407] Connection header received
0924 11:41:12.305744 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:12.305750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:12.305830 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:12.305985 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:41:12.306375 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.307493 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.308866 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.309549 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.309672 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:12.310113 (+ 441us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:41:12.312224 (+ 2111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 11:41:12.312551 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 11:41:12.312922 5820 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 11:41:12.314543 5846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:12.304829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.0.1:49762)
0924 11:41:12.305446 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:12.305461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:12.305588 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:12.305996 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:12.305999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:12.306008 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:41:12.306242 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:12.306249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.307612 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.307616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:12.308746 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:12.308756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.309653 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.309706 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:12.309818 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:12.309864 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:41:12.312444 (+ 2580us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 11:41:12.314133 (+ 1689us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1444,"thread_start_us":353,"threads_started":1}
I20260924 11:41:12.315430 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5751
I20260924 11:41:12.316391 5812 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 11:41:12.324657 5847 webserver.cc:533] Webserver: sslize failed:
E20260924 11:41:12.324754 5847 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:41:12.405160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5751
2026-09-24T11:41:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (685 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T11:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:12Z Disabled control of system clock
I20260924 11:41:12.483505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37581
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37581
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/testchainca.cert with env {}
W20260924 11:41:12.773241 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:12.773473 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:12.773545 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:12.780151 5852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:12.780283 5852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:12.780339 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:12.780428 5852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:12.780462 5852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:41:12.780514 5852 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:41:12.780548 5852 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 11:41:12.789238 5852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:12.790179 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:12.791216 5852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:12.804620 5860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:12.804644 5857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:12.804656 5858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:12.805195 5852 server_base.cc:1034] running on GCE node
I20260924 11:41:12.805962 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:12.807192 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:12.808374 5852 hybrid_clock.cc:648] HybridClock initialized: now 1790250072808336 us; error 35 us; skew 500 ppm
I20260924 11:41:12.808699 5852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:12.810675 5852 webserver.cc:460] Webserver started at http://127.0.97.62:34759/ using document root <none> and password file <none>
I20260924 11:41:12.811273 5852 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:12.811388 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:12.811702 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:12.814255 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "58bea035508d4c4f853bcdf3f7a0a13f"
format_stamp: "Formatted at 2026-09-24 11:41:12 on dist-test-slave-qhrh"
server_key: "abd5343b5c899fc31e5c8b3a6181b2fe"
server_key_iv: "02490b5976e4431a3e87816f832ea4be"
server_key_version: "encryptionkey@0"
I20260924 11:41:12.814975 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "58bea035508d4c4f853bcdf3f7a0a13f"
format_stamp: "Formatted at 2026-09-24 11:41:12 on dist-test-slave-qhrh"
server_key: "abd5343b5c899fc31e5c8b3a6181b2fe"
server_key_iv: "02490b5976e4431a3e87816f832ea4be"
server_key_version: "encryptionkey@0"
I20260924 11:41:12.819519 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:41:12.822552 5866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:12.823782 5852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:12.823961 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "58bea035508d4c4f853bcdf3f7a0a13f"
format_stamp: "Formatted at 2026-09-24 11:41:12 on dist-test-slave-qhrh"
server_key: "abd5343b5c899fc31e5c8b3a6181b2fe"
server_key_iv: "02490b5976e4431a3e87816f832ea4be"
server_key_version: "encryptionkey@0"
I20260924 11:41:12.824170 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:12.840695 5852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:12.841387 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:12.841637 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:12.858620 5852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37581
I20260924 11:41:12.858650 5917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37581 every 8 connection(s)
I20260924 11:41:12.859752 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:12.863125 5918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:12.867445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5852
I20260924 11:41:12.867654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:12.868460 388 external_mini_cluster.cc:1428] Setting key 81ff1e1176a3b5e93476a1104bab98d4
I20260924 11:41:12.873503 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f: Bootstrap starting.
I20260924 11:41:12.876600 5918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:12.877619 5918 log.cc:824] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:12.880543 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f: No bootstrap required, opened a new log
I20260924 11:41:12.881574 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:12.870129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.97.62:37581)
0924 11:41:12.871127 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:12.871136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:12.871175 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:41:12.872822 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:12.872848 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:12.873232 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:12.873505 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:12.874060 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.875269 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.877128 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.877769 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.877904 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:12.879111 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:12.879121 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:12.879135 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:12.879759 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:12.879852 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:12.879874 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:12.879989 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:12.880183 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 11:41:12.880466 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":336,"threads_started":1}
I20260924 11:41:12.884953 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:41:12.888477 5918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37581 } }
I20260924 11:41:12.888783 5918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:12.888892 5918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58bea035508d4c4f853bcdf3f7a0a13f, State: Initialized, Role: FOLLOWER
I20260924 11:41:12.889384 5918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37581 } }
I20260924 11:41:12.889602 5918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:12.889739 5918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:12.889904 5918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:12.892014 5918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37581 } }
I20260924 11:41:12.892484 5918 leader_election.cc:302] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58bea035508d4c4f853bcdf3f7a0a13f; no voters:
I20260924 11:41:12.892961 5918 leader_election.cc:288] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:12.893101 5923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:12.893841 5923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [term 1 LEADER]: Becoming Leader. State: Replica: 58bea035508d4c4f853bcdf3f7a0a13f, State: Running, Role: LEADER
I20260924 11:41:12.894299 5923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37581 } }
I20260924 11:41:12.894877 5918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:12.897113 5924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37581 } } }
I20260924 11:41:12.897419 5924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:12.897619 5925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58bea035508d4c4f853bcdf3f7a0a13f. Latest consensus state: current_term: 1 leader_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bea035508d4c4f853bcdf3f7a0a13f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37581 } } }
I20260924 11:41:12.898499 5925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:12.904955 5931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:12.908743 5931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:12.915635 5931 catalog_manager.cc:1347] Generated new cluster ID: 458b104ed6294d57ab7d9e0ffca22c54
I20260924 11:41:12.915750 5931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:12.934051 5931 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:12.935763 5931 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:12.942399 5931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58bea035508d4c4f853bcdf3f7a0a13f: Generated new TSK 0
I20260924 11:41:12.943245 5931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:41:13.016054 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:12.956318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.0.97.62:37581)
0924 11:41:12.956528 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:12.956538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:12.956935 (+ 397us) server_negotiation.cc:407] Connection header received
0924 11:41:12.957129 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:12.957133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:12.957195 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:12.957303 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:41:12.957869 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.958533 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.960415 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.960801 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.960887 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:12.961357 (+ 470us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:41:13.015292 (+ 53935us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 11:41:13.015715 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55664: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:41:13.016211 5921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55664: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 11:41:13.016316 5947 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:12.956232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37581 (local address 127.0.0.1:55664)
0924 11:41:12.956823 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:12.956843 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:12.956992 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:12.957347 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:12.957352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:12.957364 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:41:12.957712 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:12.957742 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.958663 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.958668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:12.960299 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:12.960312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:12.960907 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:12.960952 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:12.961115 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:12.961155 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:41:13.015591 (+ 54436us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 11:41:13.015897 (+ 306us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20260924 11:41:13.017264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5852
I20260924 11:41:13.018781 5913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:13.108573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5852
2026-09-24T11:41:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (683 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (373 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (372 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:13 dist-test-slave-qhrh krb5kdc[5962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:13 dist-test-slave-qhrh krb5kdc[5962](info): set up 2 sockets
Sep 24 11:41:13 dist-test-slave-qhrh krb5kdc[5962](info): commencing operation
krb5kdc: starting...
W20260924 11:41:14.884405 388 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:14 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:41:15.064306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36749
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:15.354727 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:15.354972 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:15.355126 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:15.361687 5975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:15.361774 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:15.361881 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:15.361991 5975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:15.362052 5975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:15.371270 5975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:15.372414 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:15.373653 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:15.384667 5980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:15.384716 5981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:15.384621 5983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:15.385627 5975 server_base.cc:1034] running on GCE node
I20260924 11:41:15.386307 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:15.387567 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:15.388739 5975 hybrid_clock.cc:648] HybridClock initialized: now 1790250075388693 us; error 51 us; skew 500 ppm
I20260924 11:41:15.389036 5975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:15 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:15.391829 5975 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:41:15.393255 5975 webserver.cc:460] Webserver started at http://127.0.97.62:43979/ using document root <none> and password file <none>
I20260924 11:41:15.393872 5975 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:15.394037 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:15.394328 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:15.396989 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "d6339d4121784aa095e45ab8c12ffbd0"
format_stamp: "Formatted at 2026-09-24 11:41:15 on dist-test-slave-qhrh"
server_key: "145e1f20ca242447c9115b22baab0244"
server_key_iv: "7e8d308c769ab60e676fe1969b0d7c3b"
server_key_version: "encryptionkey@0"
I20260924 11:41:15.397745 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "d6339d4121784aa095e45ab8c12ffbd0"
format_stamp: "Formatted at 2026-09-24 11:41:15 on dist-test-slave-qhrh"
server_key: "145e1f20ca242447c9115b22baab0244"
server_key_iv: "7e8d308c769ab60e676fe1969b0d7c3b"
server_key_version: "encryptionkey@0"
I20260924 11:41:15.402438 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 11:41:15.405439 5990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:15.406596 5975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 11:41:15.406790 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "d6339d4121784aa095e45ab8c12ffbd0"
format_stamp: "Formatted at 2026-09-24 11:41:15 on dist-test-slave-qhrh"
server_key: "145e1f20ca242447c9115b22baab0244"
server_key_iv: "7e8d308c769ab60e676fe1969b0d7c3b"
server_key_version: "encryptionkey@0"
I20260924 11:41:15.406983 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:15.429014 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:15.432430 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:15.432667 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:15.449246 5975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36749
I20260924 11:41:15.449345 6041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36749 every 8 connection(s)
I20260924 11:41:15.450428 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:15.453471 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:15.459805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 5975
I20260924 11:41:15.460017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:15.460801 388 external_mini_cluster.cc:1428] Setting key 3e74350ae00e0e6de33b71089081286e
I20260924 11:41:15.463702 6042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0: Bootstrap starting.
I20260924 11:41:15.466895 6042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:15.468020 6042 log.cc:824] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:15.470538 6042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0: No bootstrap required, opened a new log
Sep 24 11:41:15 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:15.478711 6042 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6339d4121784aa095e45ab8c12ffbd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260924 11:41:15.479102 6042 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:15.479251 6042 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6339d4121784aa095e45ab8c12ffbd0, State: Initialized, Role: FOLLOWER
I20260924 11:41:15.479811 6042 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6339d4121784aa095e45ab8c12ffbd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260924 11:41:15.480050 6042 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:15.480228 6042 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:15.480391 6042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:15.481933 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:15.462654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.97.62:36749)
0924 11:41:15.463352 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:15.463362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:15.463402 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:41:15.465124 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:15.465143 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:15.465471 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:15.465780 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:15.467494 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:15.468842 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:15.470058 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:15.470717 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:15.470899 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:15.474894 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:15.474903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:15.474915 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:15.474967 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:15.478159 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:15.478892 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:15.478897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:15.478902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:15.479068 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:15.479488 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:15.479493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:15.479497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:15.479956 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:15.480271 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:15.480644 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 11:41:15.481034 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":346,"threads_started":1}
I20260924 11:41:15.482728 6042 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6339d4121784aa095e45ab8c12ffbd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260924 11:41:15.483316 6042 leader_election.cc:302] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6339d4121784aa095e45ab8c12ffbd0; no voters:
I20260924 11:41:15.483892 6042 leader_election.cc:288] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:15.484015 6047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:15.485232 6047 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [term 1 LEADER]: Becoming Leader. State: Replica: d6339d4121784aa095e45ab8c12ffbd0, State: Running, Role: LEADER
I20260924 11:41:15.485675 6047 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6339d4121784aa095e45ab8c12ffbd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260924 11:41:15.485790 6042 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:15.488261 6049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6339d4121784aa095e45ab8c12ffbd0. Latest consensus state: current_term: 1 leader_uuid: "d6339d4121784aa095e45ab8c12ffbd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6339d4121784aa095e45ab8c12ffbd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } } }
I20260924 11:41:15.488667 6049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:15.488646 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6339d4121784aa095e45ab8c12ffbd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6339d4121784aa095e45ab8c12ffbd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } } }
I20260924 11:41:15.488938 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:15.493279 6053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:15.497208 6053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:15.504647 6053 catalog_manager.cc:1347] Generated new cluster ID: a83ca7b86447401cafa1b90f7343e770
I20260924 11:41:15.504949 6053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:15.527549 6053 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:15.528882 6053 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:15.535023 6053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6339d4121784aa095e45ab8c12ffbd0: Generated new TSK 0
I20260924 11:41:15.535801 6053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:41:15.592144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:15.887939 6068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:15.888234 6068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:15.888383 6068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:15.894853 6068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:15.894968 6068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:15.895164 6068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:15.903795 6068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:15.904815 6068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:15.906008 6068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:15.918023 6074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:15.918069 6076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:15.918561 6073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:15.919551 6068 server_base.cc:1034] running on GCE node
I20260924 11:41:15.920207 6068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:15.921042 6068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:15.922251 6068 hybrid_clock.cc:648] HybridClock initialized: now 1790250075922212 us; error 52 us; skew 500 ppm
I20260924 11:41:15.922533 6068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:15 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:15.925289 6068 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:41:15.926571 6068 webserver.cc:460] Webserver started at http://127.0.97.1:34681/ using document root <none> and password file <none>
I20260924 11:41:15.927210 6068 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:15.927345 6068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:15.927616 6068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:15.930403 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "c7af22ea12d84006b1e43c2fe58d9f51"
format_stamp: "Formatted at 2026-09-24 11:41:15 on dist-test-slave-qhrh"
server_key: "b904d167a4ce5efd374bf09724ba5faa"
server_key_iv: "4e9bd36012fac4a806ca7021ceb33583"
server_key_version: "encryptionkey@0"
I20260924 11:41:15.931151 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c7af22ea12d84006b1e43c2fe58d9f51"
format_stamp: "Formatted at 2026-09-24 11:41:15 on dist-test-slave-qhrh"
server_key: "b904d167a4ce5efd374bf09724ba5faa"
server_key_iv: "4e9bd36012fac4a806ca7021ceb33583"
server_key_version: "encryptionkey@0"
I20260924 11:41:15.935818 6068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:41:15.938843 6083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:15.939970 6068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:15.940126 6068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "c7af22ea12d84006b1e43c2fe58d9f51"
format_stamp: "Formatted at 2026-09-24 11:41:15 on dist-test-slave-qhrh"
server_key: "b904d167a4ce5efd374bf09724ba5faa"
server_key_iv: "4e9bd36012fac4a806ca7021ceb33583"
server_key_version: "encryptionkey@0"
I20260924 11:41:15.940371 6068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:15.961146 6068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:15.964365 6068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:15.964600 6068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:15.965378 6068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:15.966539 6068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:15.966652 6068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:15.966789 6068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:15.966862 6068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:15.997767 6068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43649
I20260924 11:41:15.997834 6195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43649 every 8 connection(s)
I20260924 11:41:15.998831 6068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:16.007673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6068
I20260924 11:41:16.007879 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:16.008737 388 external_mini_cluster.cc:1428] Setting key 932efb4d8ee474d71d61dabd0e907580
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:16.020315 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.001058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57903 (local address 127.0.97.62:36749)
0924 11:41:16.001448 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.001452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.002291 (+ 839us) server_negotiation.cc:407] Connection header received
0924 11:41:16.003831 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.003835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.003897 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.003985 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.005361 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.006315 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.010452 (+ 4137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.010905 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.011006 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.014675 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.014679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.014682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.014708 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:16.017611 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.018248 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.018253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.018257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.018364 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.018734 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.018739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.018742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.019005 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:16.019149 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:16.019741 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 11:41:16.019929 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":218,"threads_started":1}
I20260924 11:41:16.021106 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.001559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36749 (local address 127.0.97.1:57903)
0924 11:41:16.002156 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:16.002188 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:16.003626 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:16.004214 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:16.004221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:16.004686 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:16.005220 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:16.005235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.006477 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.006484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:16.010329 (+ 3845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:16.010340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.011148 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.011157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:16.011493 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.012135 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:16.012190 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:16.014500 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:16.017728 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:16.017735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:16.017748 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:16.018101 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:16.018460 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:16.018465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:16.018468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:16.018628 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:16.019121 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:16.019128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:16.019403 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 11:41:16.019654 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 11:41:16.020081 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":244,"threads_started":1}
I20260924 11:41:16.022513 6196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36749
I20260924 11:41:16.022768 6196 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:16.023415 6196 heartbeater.cc:511] Master 127.0.97.62:36749 requested a full tablet report, sending...
I20260924 11:41:16.025260 6007 ts_manager.cc:194] Registered new tserver with Master: c7af22ea12d84006b1e43c2fe58d9f51 (127.0.97.1:43649)
I20260924 11:41:16.026619 6007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57903
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:41:16.051525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:16.342846 6202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:16.343067 6202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:16.343230 6202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:16.350065 6202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:16.350145 6202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:16.350311 6202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:16.358956 6202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:16.359918 6202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:16.361011 6202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:16.373035 6208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:16.373075 6207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:16.372956 6210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:16.376885 6202 server_base.cc:1034] running on GCE node
I20260924 11:41:16.377365 6202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:16.378085 6202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:16.379313 6202 hybrid_clock.cc:648] HybridClock initialized: now 1790250076379278 us; error 49 us; skew 500 ppm
I20260924 11:41:16.379586 6202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.382297 6202 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:41:16.383513 6202 webserver.cc:460] Webserver started at http://127.0.97.2:37923/ using document root <none> and password file <none>
I20260924 11:41:16.384049 6202 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:16.384132 6202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:16.384397 6202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:16.386834 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "49dd2bceff4a43aa88d6581f228d2e24"
format_stamp: "Formatted at 2026-09-24 11:41:16 on dist-test-slave-qhrh"
server_key: "1b7f8d30a83225271a5c1f26dca66663"
server_key_iv: "171d59b5d07bf48708af5cc473a3d0fc"
server_key_version: "encryptionkey@0"
I20260924 11:41:16.387553 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "49dd2bceff4a43aa88d6581f228d2e24"
format_stamp: "Formatted at 2026-09-24 11:41:16 on dist-test-slave-qhrh"
server_key: "1b7f8d30a83225271a5c1f26dca66663"
server_key_iv: "171d59b5d07bf48708af5cc473a3d0fc"
server_key_version: "encryptionkey@0"
I20260924 11:41:16.391937 6202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 11:41:16.394793 6217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:16.396131 6202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:41:16.396315 6202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "49dd2bceff4a43aa88d6581f228d2e24"
format_stamp: "Formatted at 2026-09-24 11:41:16 on dist-test-slave-qhrh"
server_key: "1b7f8d30a83225271a5c1f26dca66663"
server_key_iv: "171d59b5d07bf48708af5cc473a3d0fc"
server_key_version: "encryptionkey@0"
I20260924 11:41:16.396476 6202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:16.406980 6202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:16.410341 6202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:16.410553 6202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:16.411286 6202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:16.412360 6202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:16.412434 6202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:16.412509 6202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:16.412550 6202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:16.441018 6202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39305
I20260924 11:41:16.441107 6329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39305 every 8 connection(s)
I20260924 11:41:16.442036 6202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:16.446043 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6202
I20260924 11:41:16.446235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:16.446638 388 external_mini_cluster.cc:1428] Setting key 3155a71a82180f0d3076350cf68c4c49
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:16.463688 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.444197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57839 (local address 127.0.97.62:36749)
0924 11:41:16.444325 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.444328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.445440 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 11:41:16.447349 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.447354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.447435 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.447545 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.449054 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.450164 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.452223 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.452910 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.453080 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.457234 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.457240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.457245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.457289 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:16.460659 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.461529 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.461533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.461536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.461682 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.462149 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.462154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.462157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.462463 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:16.462593 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:16.463152 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 11:41:16.463345 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 11:41:16.464696 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.444543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36749 (local address 127.0.97.2:57839)
0924 11:41:16.445277 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:16.445313 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:16.447097 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:16.447749 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:16.447755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:16.448237 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:16.448870 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:16.448890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.450318 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.450323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:16.452065 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:16.452078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.453150 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.453156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:16.453423 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.454094 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:16.454120 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:16.456999 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:16.460800 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:16.460811 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:16.460828 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:16.461402 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:16.461797 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:16.461803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:16.461809 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:16.462028 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:16.462571 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:16.462576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:16.462845 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 11:41:16.463065 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 11:41:16.463477 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":384,"threads_started":1}
I20260924 11:41:16.466255 6330 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36749
I20260924 11:41:16.466543 6330 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:41:16.467257 6330 heartbeater.cc:511] Master 127.0.97.62:36749 requested a full tablet report, sending...
I20260924 11:41:16.468658 6007 ts_manager.cc:194] Registered new tserver with Master: 49dd2bceff4a43aa88d6581f228d2e24 (127.0.97.2:39305)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 11:41:16.469527 6007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:41:16.490957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:16.780197 6335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:16.780401 6335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:16.780535 6335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:16.787391 6335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:16.787496 6335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:16.787668 6335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:16.796072 6335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:16.797067 6335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:16.798172 6335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:16.809954 6341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:16.809985 6343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:16.810164 6340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:16.810645 6335 server_base.cc:1034] running on GCE node
I20260924 11:41:16.811203 6335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:16.811959 6335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:16.813135 6335 hybrid_clock.cc:648] HybridClock initialized: now 1790250076813090 us; error 47 us; skew 500 ppm
I20260924 11:41:16.813403 6335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.816232 6335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:41:16.817512 6335 webserver.cc:460] Webserver started at http://127.0.97.3:36841/ using document root <none> and password file <none>
I20260924 11:41:16.818075 6335 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:16.818208 6335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:16.818500 6335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:16.821049 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "7fce8c57d9c44aa6923dedc7dad76ee0"
format_stamp: "Formatted at 2026-09-24 11:41:16 on dist-test-slave-qhrh"
server_key: "d4c2f6ea70a279f88283c42751fc37d8"
server_key_iv: "2f831a809f95da111912d9e0699b2383"
server_key_version: "encryptionkey@0"
I20260924 11:41:16.821727 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7fce8c57d9c44aa6923dedc7dad76ee0"
format_stamp: "Formatted at 2026-09-24 11:41:16 on dist-test-slave-qhrh"
server_key: "d4c2f6ea70a279f88283c42751fc37d8"
server_key_iv: "2f831a809f95da111912d9e0699b2383"
server_key_version: "encryptionkey@0"
I20260924 11:41:16.826087 6335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 11:41:16.829147 6350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:16.830255 6335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:16.830405 6335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "7fce8c57d9c44aa6923dedc7dad76ee0"
format_stamp: "Formatted at 2026-09-24 11:41:16 on dist-test-slave-qhrh"
server_key: "d4c2f6ea70a279f88283c42751fc37d8"
server_key_iv: "2f831a809f95da111912d9e0699b2383"
server_key_version: "encryptionkey@0"
I20260924 11:41:16.830583 6335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:16.848969 6335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:16.852037 6335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:16.852291 6335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:16.853044 6335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:16.854140 6335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:16.854235 6335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:16.854364 6335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:16.854429 6335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:16.884176 6335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35953
I20260924 11:41:16.884230 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35953 every 8 connection(s)
I20260924 11:41:16.885321 6335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:16.895628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6335
I20260924 11:41:16.895826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:16.896310 388 external_mini_cluster.cc:1428] Setting key fee8dcc05a8853d2a8a9ee0d7bd61df2
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:16.903339 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.887593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59881 (local address 127.0.97.62:36749)
0924 11:41:16.887752 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.887756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.888890 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 11:41:16.890528 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.890533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.890616 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.890721 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.892012 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.892745 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.894075 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.894610 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.894736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.898554 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.898560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.898564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.898599 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:16.901112 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.901699 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.901703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.901705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.901806 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.902154 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.902157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.902159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.902387 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:16.902485 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:16.902985 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 11:41:16.903120 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:41:16.904120 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.888026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36749 (local address 127.0.97.3:59881)
0924 11:41:16.888752 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:16.888792 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:16.890291 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:16.890949 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:16.890957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:16.891376 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:16.891858 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:16.891872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.892870 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.892877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:16.893933 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:16.893943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.894814 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.894824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:16.895079 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.895751 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:16.895782 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:16.898369 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:16.901217 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:16.901223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:16.901231 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:16.901610 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:16.901891 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:16.901894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:16.901896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:16.902048 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:16.902480 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:16.902487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:16.902725 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 11:41:16.902912 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 11:41:16.903205 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":349,"threads_started":1}
I20260924 11:41:16.905282 6463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36749
I20260924 11:41:16.905508 6463 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:16.906131 6463 heartbeater.cc:511] Master 127.0.97.62:36749 requested a full tablet report, sending...
I20260924 11:41:16.907316 6007 ts_manager.cc:194] Registered new tserver with Master: 7fce8c57d9c44aa6923dedc7dad76ee0 (127.0.97.3:35953)
I20260924 11:41:16.908062 6007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59881
I20260924 11:41:16.910954 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/test-admin.keytab.
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.934700 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 11:41:16.938935 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:16.936896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.97.1:43649)
0924 11:41:16.937471 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.937477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.937502 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:41:16.937615 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.937622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.937796 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.937905 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.938620 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39082: BlockingRecv error: recv got EOF from 127.0.0.1:39082 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":320,"threads_started":1}
W20260924 11:41:16.939411 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.940907 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.942176 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:16.955155 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.943602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.97.1:43649)
0924 11:41:16.943764 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.943769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.943928 (+ 159us) server_negotiation.cc:407] Connection header received
0924 11:41:16.944048 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.944051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.944095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.944244 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.945107 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.945999 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.947124 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.947601 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.947688 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.950731 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.950740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.950751 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.950776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:16.953114 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.953632 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.953638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.953640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.953739 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.954019 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.954021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.954023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.954308 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:16.954512 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:16.954716 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 11:41:16.954940 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 11:41:16.959697 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:16.958513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.97.1:43649)
0924 11:41:16.958674 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.958678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.958747 (+ 69us) server_negotiation.cc:407] Connection header received
0924 11:41:16.958835 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.958837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.958885 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.958949 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.959518 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39114: BlockingRecv error: recv got EOF from 127.0.0.1:39114 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 11:41:16.959817 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.961131 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.962273 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:16.973819 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.963239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.97.1:43649)
0924 11:41:16.963393 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.963397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.963542 (+ 145us) server_negotiation.cc:407] Connection header received
0924 11:41:16.963627 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.963630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.963688 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.963755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.964554 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.965148 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.966262 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.966663 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.966748 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.969801 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.969805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.969808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.969857 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:16.971902 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.972496 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.972500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.972502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.972593 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.972875 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.972878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.972880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.973085 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:16.973249 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:16.973415 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 11:41:16.973574 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:41:16.977154 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:16.975776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.0.97.1:43649)
0924 11:41:16.975944 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.975949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.976186 (+ 237us) server_negotiation.cc:407] Connection header received
0924 11:41:16.976288 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.976291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.976344 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.976413 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.976968 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39132: BlockingRecv error: recv got EOF from 127.0.0.1:39132 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:41:16.977324 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250027667721-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.978580 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:16.979722 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:41:16 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:16.991255 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.980823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.0.97.1:43649)
0924 11:41:16.981001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.981006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.981162 (+ 156us) server_negotiation.cc:407] Connection header received
0924 11:41:16.981261 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.981263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.981302 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.981366 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.982137 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.982675 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.983680 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.984062 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.984183 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.987075 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.987079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.987082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.987106 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:16.989306 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.989811 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.989814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.989816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.989906 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:16.990169 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:16.990172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:16.990174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:16.990375 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:16.990528 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:16.990787 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 11:41:16.990957 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:41:16.992451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6068
I20260924 11:41:17.003605 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:16.993034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.97.1:43649)
0924 11:41:16.993203 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:16.993208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:16.993379 (+ 171us) server_negotiation.cc:407] Connection header received
0924 11:41:16.993463 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:16.993466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:16.993510 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:16.993575 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:16.994308 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.994982 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.996180 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:16.996564 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:16.996648 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:16.998420 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:16.998426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:16.998430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:16.998459 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:17.001437 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:17.001960 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:17.001965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:17.001968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:17.002096 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:17.002398 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:17.002402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:17.002406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:17.002658 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:17.002904 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:17.003082 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 11:41:17.003302 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:41:17.004248 6191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:17.085031 6196 heartbeater.cc:503] Master 127.0.97.62:36749 was elected leader, sending a full tablet report...
I20260924 11:41:17.085675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6068
W20260924 11:41:17.101424 5994 connection.cc:535] server connection from 127.0.97.1:57903 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:41:17.101711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6202
Sep 24 11:41:17 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:41:17.115546 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:17.102455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.0.97.2:39305)
0924 11:41:17.102977 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:17.102985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:17.103008 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:41:17.103059 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:17.103065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:17.103259 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:17.103376 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:17.104195 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:17.105118 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:17.106260 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:17.106794 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:17.106886 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:17.110655 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:17.110664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:17.110677 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:17.110704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:17.113443 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:17.114051 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:17.114057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:17.114059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:17.114176 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:17.114460 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:17.114462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:17.114476 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:17.114751 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:17.114963 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:17.115146 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 11:41:17.115303 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260924 11:41:17.116719 6325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:17.193696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6202
I20260924 11:41:17.208675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6335
Sep 24 11:41:17 dist-test-slave-qhrh krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:41:17.232007 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:17.209491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.97.3:35953)
0924 11:41:17.210126 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:17.210133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:17.210158 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:41:17.210210 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:17.210217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:17.210421 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:17.210612 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:17.211706 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:17.212780 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:17.214003 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:17.214537 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:17.214687 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:17.227065 (+ 12378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:17.227076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:17.227089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:17.227122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:17.229589 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:17.230214 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:17.230220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:17.230222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:17.230326 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:17.230662 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:17.230669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:17.230672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:17.230971 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:17.231260 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:17.231519 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 11:41:17.231755 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":302,"threads_started":1}
I20260924 11:41:17.233178 6458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:17.310073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6335
I20260924 11:41:17.325438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 5975
I20260924 11:41:17.326524 6037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:17.420445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 5975
2026-09-24T11:41:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3571 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:17 dist-test-slave-qhrh krb5kdc[6490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:17 dist-test-slave-qhrh krb5kdc[6490](info): set up 2 sockets
Sep 24 11:41:17 dist-test-slave-qhrh krb5kdc[6490](info): commencing operation
krb5kdc: starting...
W20260924 11:41:18.452531 388 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:18 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 11:41:18.629637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:18.919832 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:18.920109 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:18.920264 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:18.927063 6503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:18.927188 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:18.927291 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:18.927389 6503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:18.927474 6503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:18.936277 6503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:18.937355 6503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:18.938550 6503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:18.949210 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:18.949222 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:18.949236 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:18.950359 6503 server_base.cc:1034] running on GCE node
I20260924 11:41:18.950994 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:18.952100 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:18.953320 6503 hybrid_clock.cc:648] HybridClock initialized: now 1790250078953286 us; error 27 us; skew 500 ppm
I20260924 11:41:18.953615 6503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:18 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:18.956375 6503 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 11:41:18.957607 6503 webserver.cc:460] Webserver started at http://127.0.97.62:35995/ using document root <none> and password file <none>
I20260924 11:41:18.958168 6503 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:18.958294 6503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:18.958629 6503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:18.961750 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "57cdef02ecfa4c998ae8a9e3f5c41170"
format_stamp: "Formatted at 2026-09-24 11:41:18 on dist-test-slave-qhrh"
server_key: "abdd4ba4beab2001d20e54ea760e390f"
server_key_iv: "3e210acd8d50c002c26cc991a6ea11ae"
server_key_version: "encryptionkey@0"
I20260924 11:41:18.962455 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "57cdef02ecfa4c998ae8a9e3f5c41170"
format_stamp: "Formatted at 2026-09-24 11:41:18 on dist-test-slave-qhrh"
server_key: "abdd4ba4beab2001d20e54ea760e390f"
server_key_iv: "3e210acd8d50c002c26cc991a6ea11ae"
server_key_version: "encryptionkey@0"
I20260924 11:41:18.967065 6503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:41:18.970045 6518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:18.971139 6503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:18.971304 6503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "57cdef02ecfa4c998ae8a9e3f5c41170"
format_stamp: "Formatted at 2026-09-24 11:41:18 on dist-test-slave-qhrh"
server_key: "abdd4ba4beab2001d20e54ea760e390f"
server_key_iv: "3e210acd8d50c002c26cc991a6ea11ae"
server_key_version: "encryptionkey@0"
I20260924 11:41:18.971484 6503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:18.988329 6503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:18.991583 6503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:18.991824 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:19.008600 6503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44733
I20260924 11:41:19.008621 6569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44733 every 8 connection(s)
I20260924 11:41:19.009800 6503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:19.014420 6570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:19.015590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6503
I20260924 11:41:19.015799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:19.016575 388 external_mini_cluster.cc:1428] Setting key 81f7618e94810a2bf8247ec05c241325
I20260924 11:41:19.028931 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Bootstrap starting.
Sep 24 11:41:19 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:19.032529 6570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:19.033470 6570 log.cc:824] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:19.035970 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: No bootstrap required, opened a new log
I20260924 11:41:19.039467 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:19.019272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.0.97.62:44733)
0924 11:41:19.020582 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:19.020592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:19.020627 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:41:19.022067 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:19.022084 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:19.022393 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:19.022665 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:19.024735 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:19.025940 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:19.027693 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:19.028405 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:19.028536 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:19.032918 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:19.032925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:19.032937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:19.032965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:19.035803 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:19.036546 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:19.036551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:19.036557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:19.036702 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:19.037076 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:19.037080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:19.037082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:19.037554 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:19.037867 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:19.038250 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 11:41:19.038594 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":902,"threads_started":1}
I20260924 11:41:19.043316 6570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } }
I20260924 11:41:19.043596 6570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:19.043684 6570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cdef02ecfa4c998ae8a9e3f5c41170, State: Initialized, Role: FOLLOWER
I20260924 11:41:19.044198 6570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } }
I20260924 11:41:19.044442 6570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:19.044587 6570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:19.044775 6570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:19.046695 6570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } }
I20260924 11:41:19.047149 6570 leader_election.cc:302] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57cdef02ecfa4c998ae8a9e3f5c41170; no voters:
I20260924 11:41:19.047677 6570 leader_election.cc:288] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:19.047753 6575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:19.048429 6575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 LEADER]: Becoming Leader. State: Replica: 57cdef02ecfa4c998ae8a9e3f5c41170, State: Running, Role: LEADER
I20260924 11:41:19.048956 6575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } }
I20260924 11:41:19.049535 6570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:19.051532 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } }
I20260924 11:41:19.051985 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:19.051563 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cdef02ecfa4c998ae8a9e3f5c41170. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } }
I20260924 11:41:19.052565 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:19.052758 6584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:19.057312 6584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:19.065461 6584 catalog_manager.cc:1347] Generated new cluster ID: e47f00f057ad4e95bb918af8deb82a2b
I20260924 11:41:19.065577 6584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:19.075419 6584 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:19.076866 6584 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:19.083837 6584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Generated new TSK 0
I20260924 11:41:19.084503 6584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 11:41:19.136072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44733
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:19.431294 6596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:19.431524 6596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:19.431690 6596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:19.438344 6596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:19.438427 6596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:19.438596 6596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:19.446768 6596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:19.447831 6596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:19.449012 6596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:19.461658 6604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:19.461620 6602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:19.461692 6601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:19.462256 6596 server_base.cc:1034] running on GCE node
I20260924 11:41:19.462791 6596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:19.463615 6596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:19.464823 6596 hybrid_clock.cc:648] HybridClock initialized: now 1790250079464773 us; error 40 us; skew 500 ppm
I20260924 11:41:19.465130 6596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:19 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:19.468026 6596 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 11:41:19.469282 6596 webserver.cc:460] Webserver started at http://127.0.97.1:41289/ using document root <none> and password file <none>
I20260924 11:41:19.469907 6596 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:19.470057 6596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:19.470341 6596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:19.473187 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "834abac14e68482495726b4d94f6f5d5"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "252a2c69058e90ef1ea2b76c98f49d6f"
server_key_iv: "339428389b15b3aad945517964652070"
server_key_version: "encryptionkey@0"
I20260924 11:41:19.473937 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "834abac14e68482495726b4d94f6f5d5"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "252a2c69058e90ef1ea2b76c98f49d6f"
server_key_iv: "339428389b15b3aad945517964652070"
server_key_version: "encryptionkey@0"
I20260924 11:41:19.478799 6596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:41:19.481858 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:19.483178 6596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:41:19.483307 6596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "834abac14e68482495726b4d94f6f5d5"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "252a2c69058e90ef1ea2b76c98f49d6f"
server_key_iv: "339428389b15b3aad945517964652070"
server_key_version: "encryptionkey@0"
I20260924 11:41:19.483462 6596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:19.510819 6596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:19.514189 6596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:19.514413 6596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:19.515172 6596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:19.516311 6596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:19.516400 6596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:19.516475 6596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:19.516533 6596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:19.547266 6596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39837
I20260924 11:41:19.547343 6723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39837 every 8 connection(s)
I20260924 11:41:19.548448 6596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:19.552304 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6596
I20260924 11:41:19.552493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:19.553164 388 external_mini_cluster.cc:1428] Setting key 0f0006432fa4bac534889d46b2deb745
Sep 24 11:41:19 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:19.569593 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:19.550807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47785 (local address 127.0.97.62:44733)
0924 11:41:19.551164 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:19.551168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:19.552055 (+ 887us) server_negotiation.cc:407] Connection header received
0924 11:41:19.554184 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:19.554189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:19.554275 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:19.554387 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:19.556061 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:19.556818 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:19.559201 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:19.559771 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:19.559877 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:19.563302 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:19.563310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:19.563318 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:19.563356 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:19.566638 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:19.567291 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:19.567298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:19.567305 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:19.567438 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:19.567816 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:19.567824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:19.567830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:19.568213 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:19.568382 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:19.568899 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 11:41:19.569147 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":207,"threads_started":1}
I20260924 11:41:19.570000 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:19.551299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:47785)
0924 11:41:19.551882 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:19.551919 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:19.553937 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:19.554591 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:19.554599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:19.555168 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:19.555871 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:19.555894 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:19.556955 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:19.556964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:19.559075 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:19.559090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:19.559963 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:19.559971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:19.560286 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:19.560949 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:19.560996 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:19.563120 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:19.566794 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:19.566800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:19.566809 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:19.567178 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:19.567547 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:19.567550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:19.567552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:19.567707 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:19.568334 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:19.568339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:19.568589 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 11:41:19.568778 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 11:41:19.569073 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":233,"threads_started":1}
I20260924 11:41:19.571544 6724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44733
I20260924 11:41:19.571811 6724 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:19.572453 6724 heartbeater.cc:511] Master 127.0.97.62:44733 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:41:19.574415 6535 ts_manager.cc:194] Registered new tserver with Master: 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837)
I20260924 11:41:19.575832 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 11:41:19.596570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44733
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:19.891500 6730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:19.891773 6730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:19.891911 6730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:19.898495 6730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:19.898634 6730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:19.898797 6730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:19.907239 6730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:19.908392 6730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:19.909554 6730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:19.922154 6736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:19.922164 6738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:19.922148 6735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:19.922875 6730 server_base.cc:1034] running on GCE node
I20260924 11:41:19.923441 6730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:19.924305 6730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:19.925511 6730 hybrid_clock.cc:648] HybridClock initialized: now 1790250079925473 us; error 47 us; skew 500 ppm
I20260924 11:41:19.925804 6730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:19 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:19.928665 6730 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 11:41:19.929963 6730 webserver.cc:460] Webserver started at http://127.0.97.2:40879/ using document root <none> and password file <none>
I20260924 11:41:19.930578 6730 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:19.930727 6730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:19.931002 6730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:19.933633 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "99913db5b7e9468daf0f29cbb0971b4a"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "ea037925ffe6f38b545a56dd73485644"
server_key_iv: "c0a72a50060d5a8f475dfa5b283f9090"
server_key_version: "encryptionkey@0"
I20260924 11:41:19.934355 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "99913db5b7e9468daf0f29cbb0971b4a"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "ea037925ffe6f38b545a56dd73485644"
server_key_iv: "c0a72a50060d5a8f475dfa5b283f9090"
server_key_version: "encryptionkey@0"
I20260924 11:41:19.939000 6730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:41:19.941963 6745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:19.943094 6730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:41:19.943241 6730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "99913db5b7e9468daf0f29cbb0971b4a"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "ea037925ffe6f38b545a56dd73485644"
server_key_iv: "c0a72a50060d5a8f475dfa5b283f9090"
server_key_version: "encryptionkey@0"
I20260924 11:41:19.943397 6730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:19.961391 6730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:19.964848 6730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:19.965102 6730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:19.965924 6730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:19.967125 6730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:19.967228 6730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:19.967365 6730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:19.967437 6730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:19.998489 6730 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37021
I20260924 11:41:19.998534 6857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37021 every 8 connection(s)
I20260924 11:41:19.999614 6730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:20.001176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6730
I20260924 11:41:20.001368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:20.001782 388 external_mini_cluster.cc:1428] Setting key c029530fd5ccd9a17e707cf759627c6e
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:20.023386 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.001898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49107 (local address 127.0.97.62:44733)
0924 11:41:20.002073 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.002078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.003352 (+ 1274us) server_negotiation.cc:407] Connection header received
0924 11:41:20.005169 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.005177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.005261 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.005361 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:20.006814 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.007968 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.010139 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.010651 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.010743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.015138 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:20.015144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:20.015150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:20.015186 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:20.018699 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.019378 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.019383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.019387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.019557 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.020343 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.020351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.020357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.021003 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:20.021244 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:20.022657 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0924 11:41:20.022993 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:41:20.025209 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.002402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:49107)
0924 11:41:20.003210 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:20.003264 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:20.004913 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:20.005543 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:20.005550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:20.006072 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:20.006632 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.006649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.008136 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.008158 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.009981 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.009996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.010875 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.010882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.011167 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.011932 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:20.011965 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:20.014916 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:20.018840 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.018847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.018858 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.019247 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.019727 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.019737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.019744 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.020107 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.021371 (+ 1264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:20.021380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:20.022051 (+ 671us) client_negotiation.cc:769] Sending connection context
0924 11:41:20.023700 (+ 1649us) client_negotiation.cc:240] Negotiation successful
0924 11:41:20.024178 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260924 11:41:20.026804 6858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44733
I20260924 11:41:20.027103 6858 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:20.027952 6858 heartbeater.cc:511] Master 127.0.97.62:44733 requested a full tablet report, sending...
I20260924 11:41:20.029481 6535 ts_manager.cc:194] Registered new tserver with Master: 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021)
I20260924 11:41:20.030258 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 11:41:20.051755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44733
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:20.349521 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:20.349735 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:20.349808 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:20.356794 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:20.356879 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:20.356992 6863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:20.365566 6863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:20.366546 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:20.367671 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:20.380316 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:20.380285 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:20.380381 6871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:20.381343 6863 server_base.cc:1034] running on GCE node
I20260924 11:41:20.381816 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:20.382597 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:20.383805 6863 hybrid_clock.cc:648] HybridClock initialized: now 1790250080383776 us; error 50 us; skew 500 ppm
I20260924 11:41:20.384099 6863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250080, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:20.387053 6863 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 11:41:20.388387 6863 webserver.cc:460] Webserver started at http://127.0.97.3:42377/ using document root <none> and password file <none>
I20260924 11:41:20.388931 6863 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:20.389042 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:20.389254 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:20.391718 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "5d468aa0894d49cf8c3ebb1997aaa21f"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "9cc5d87ba37322e583966a59b1822cdb"
server_key_iv: "64485fc036ea9eb90d828d483e8fa8c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:20.392446 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5d468aa0894d49cf8c3ebb1997aaa21f"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "9cc5d87ba37322e583966a59b1822cdb"
server_key_iv: "64485fc036ea9eb90d828d483e8fa8c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:20.396874 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:41:20.399756 6878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:20.400918 6863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:20.401075 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "5d468aa0894d49cf8c3ebb1997aaa21f"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "9cc5d87ba37322e583966a59b1822cdb"
server_key_iv: "64485fc036ea9eb90d828d483e8fa8c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:20.401243 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:20.416325 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:20.420292 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:20.420555 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:20.421361 6863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:20.422483 6863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:20.422567 6863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:20.422737 6863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:20.422786 6863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:20.453270 6863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41155
I20260924 11:41:20.453344 6990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41155 every 8 connection(s)
I20260924 11:41:20.454391 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:20.459151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6863
I20260924 11:41:20.459343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:20.459790 388 external_mini_cluster.cc:1428] Setting key b6eff251895908cfa9bc40739ba806f1
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250080, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:20.472565 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.456816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50191 (local address 127.0.97.62:44733)
0924 11:41:20.457032 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.457051 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.458361 (+ 1310us) server_negotiation.cc:407] Connection header received
0924 11:41:20.460382 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.460386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.460477 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.460616 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:20.462307 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.463148 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.464596 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.464963 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.465057 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.467999 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:20.468004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:20.468007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:20.468037 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:20.470259 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.470835 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.470839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.470841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.470949 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.471278 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.471281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.471283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.471519 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:20.471623 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:20.472186 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 11:41:20.472335 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 11:41:20.473409 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.457342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:50191)
0924 11:41:20.458221 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:20.458257 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:20.460083 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:20.460830 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:20.460839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:20.461427 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:20.462139 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.462162 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.463285 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.463289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.464489 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.464499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.465133 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.465137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.465307 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.465847 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:20.465871 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:20.467833 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:20.470369 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.470374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.470383 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.470742 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.471041 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.471044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.471046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.471193 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.471619 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:20.471624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:20.471867 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 11:41:20.472064 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 11:41:20.472409 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":401,"threads_started":1}
I20260924 11:41:20.474539 6991 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44733
I20260924 11:41:20.474817 6991 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:20.475414 6991 heartbeater.cc:511] Master 127.0.97.62:44733 requested a full tablet report, sending...
I20260924 11:41:20.476624 6534 ts_manager.cc:194] Registered new tserver with Master: 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155)
I20260924 11:41:20.477401 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50191
I20260924 11:41:20.486328 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260924 11:41:20.525961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42233
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44733,127.0.97.61:42233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:41:20.578560 6724 heartbeater.cc:503] Master 127.0.97.62:44733 was elected leader, sending a full tablet report...
W20260924 11:41:20.831390 6996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:20.831637 6996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:20.831718 6996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:20.838433 6996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:20.838518 6996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:20.838567 6996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:20.838639 6996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:20.838719 6996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:20.848101 6996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44733,127.0.97.61:42233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
W20260924 11:41:20.848410 6996 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:44733,127.0.97.61:42233'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 11:41:20.849323 6996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:20.850627 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:20.861719 7001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:20.861742 7002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:20.861704 7004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:20.862396 6996 server_base.cc:1034] running on GCE node
I20260924 11:41:20.863123 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:20.864359 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:20.865582 6996 hybrid_clock.cc:648] HybridClock initialized: now 1790250080865514 us; error 44 us; skew 500 ppm
I20260924 11:41:20.865864 6996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250080, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:20.868846 6996 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 11:41:20.870226 6996 webserver.cc:460] Webserver started at http://127.0.97.61:39307/ using document root <none> and password file <none>
I20260924 11:41:20.870854 6996 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:20.870967 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:20.871239 6996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:20.873927 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/instance:
uuid: "8898367107cd49fb8bd6dccff975f1d3"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "04dce6dbe5d98699d1bfef249b482873"
server_key_iv: "05071f2bf458fafa4f1f4ef121aee5a7"
server_key_version: "encryptionkey@0"
I20260924 11:41:20.874681 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal/instance:
uuid: "8898367107cd49fb8bd6dccff975f1d3"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "04dce6dbe5d98699d1bfef249b482873"
server_key_iv: "05071f2bf458fafa4f1f4ef121aee5a7"
server_key_version: "encryptionkey@0"
I20260924 11:41:20.879335 6996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:41:20.882316 7011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:20.883407 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:20.883567 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
uuid: "8898367107cd49fb8bd6dccff975f1d3"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "04dce6dbe5d98699d1bfef249b482873"
server_key_iv: "05071f2bf458fafa4f1f4ef121aee5a7"
server_key_version: "encryptionkey@0"
I20260924 11:41:20.883749 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:20.902993 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:20.906622 6996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:20.906863 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:20.923609 6996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42233
I20260924 11:41:20.923638 7062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42233 every 8 connection(s)
I20260924 11:41:20.924811 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
I20260924 11:41:20.927965 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:20.930424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 6996
I20260924 11:41:20.935139 7063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250080, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:20.951519 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.937811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.0.97.62:44733)
0924 11:41:20.937991 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.937999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.938477 (+ 478us) server_negotiation.cc:407] Connection header received
0924 11:41:20.938671 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.938675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.938732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.938828 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:20.939765 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.940485 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.941947 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.942310 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.942437 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.945487 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:20.945493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:20.945497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:20.945530 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:20.948809 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.949392 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.949399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.949404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.949518 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.949826 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.949831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.949834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.950213 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:20.950395 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:20.951014 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 11:41:20.951197 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:41:20.951884 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.937785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:43594)
0924 11:41:20.938357 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:20.938385 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:20.938547 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:20.938874 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:20.938879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:20.939420 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:20.939629 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.939636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.940625 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.940631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.941806 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.941817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.942503 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.942511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.942674 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.943198 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:20.943216 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:20.945300 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:20.948909 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.948916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.948920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.949287 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.949616 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.949618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.949621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.949729 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.950445 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:20.950454 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:20.950652 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 11:41:20.950920 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 11:41:20.951193 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
I20260924 11:41:20.951937 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.932534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.0.97.61:42233)
0924 11:41:20.933272 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.933285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.933324 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:41:20.934831 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.934849 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.935163 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.935443 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:20.937146 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.938884 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.940037 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.940818 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.940999 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.944740 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:20.944748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:20.944762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:20.944804 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:20.948219 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.948812 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.948819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.948825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.948996 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.949338 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.949345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.949350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.949889 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:20.950131 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:20.950525 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 11:41:20.950920 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":360,"threads_started":1}
I20260924 11:41:20.954062 7063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:41:20 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250080, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 11:41:20.964304 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.954661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55438)
0924 11:41:20.954814 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:20.954827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:20.954889 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:20.955143 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:20.955146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:20.955396 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:20.955504 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.955510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.956402 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.956405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.957247 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.957256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.957845 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.957848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:20.958020 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.958470 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:20.958483 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:20.960299 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:20.962608 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.962611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.962613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.962899 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.963156 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:20.963159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:20.963161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:20.963258 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:20.963648 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:20.963650 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:20.963754 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 11:41:20.963856 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:41:20.964014 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 11:41:20.964372 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.954819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.0.97.61:42233)
0924 11:41:20.954955 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.954959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.954971 (+ 12us) server_negotiation.cc:407] Connection header received
0924 11:41:20.955002 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.955005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.955047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.955115 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:20.955613 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.956305 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.957350 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.957735 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.957827 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:20.960481 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:20.960485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:20.960488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:20.960512 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:20.962503 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.962981 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.962984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.962986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.963078 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:20.963342 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:20.963345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:20.963346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:20.963560 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:20.963652 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:20.963931 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 11:41:20.964106 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":46}
I20260924 11:41:20.968358 7063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Bootstrap starting.
I20260924 11:41:20.970945 7063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:20.971833 7063 log.cc:824] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:20.974037 7063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: No bootstrap required, opened a new log
I20260924 11:41:20.980463 7063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } }
I20260924 11:41:20.980710 7063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:20.980774 7063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8898367107cd49fb8bd6dccff975f1d3, State: Initialized, Role: FOLLOWER
I20260924 11:41:20.981240 7063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } }
I20260924 11:41:20.981853 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } } }
I20260924 11:41:20.982117 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:41:20.982590 7063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:41:20.990150 7080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:41:20.990263 7080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:41:20.992298 6996 master_runner.cc:428] Detected that this master 8898367107cd49fb8bd6dccff975f1d3 is joining an existing cluster
I20260924 11:41:20.992388 6996 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:42233
I20260924 11:41:21.010154 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.995625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55454)
0924 11:41:20.996587 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:20.996605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:20.996753 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:20.997280 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:20.997284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:20.997966 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:20.998329 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:20.998337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.999245 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:20.999249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.000119 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.000129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.001429 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.001435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.001585 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.002151 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.002170 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.003142 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.007668 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.007674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.007680 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.008223 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.008564 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.008568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.008573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.008719 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.009323 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.009327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.009440 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.009555 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.009775 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":693,"threads_started":1}
I20260924 11:41:21.011026 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.995680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.97.61:42233)
0924 11:41:20.995899 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.995908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.996677 (+ 769us) server_negotiation.cc:407] Connection header received
0924 11:41:20.996878 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.996889 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.996972 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.997106 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:20.998439 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:20.999090 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.000284 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.000990 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.001113 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.003292 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.003297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.003302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.003329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.007527 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.008328 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.008333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.008337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.008440 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.008813 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.008818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.008821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.009128 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.009269 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.010538 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.010743 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:41:21.022745 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.995425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.0.97.62:44733)
0924 11:41:20.995873 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:20.995879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:20.997574 (+ 1695us) server_negotiation.cc:407] Connection header received
0924 11:41:20.997726 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:20.997731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:20.997809 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:20.997884 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.001286 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.002246 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.004119 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.004802 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.004915 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.007494 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.007500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.007504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.007537 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.017266 (+ 9729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.020909 (+ 3643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.020914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.020918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.021054 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.021425 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.021430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.021433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.021686 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.021833 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.022221 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.022432 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260924 11:41:21.023023 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:20.996721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:43598)
0924 11:41:20.997459 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:20.997476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:20.997603 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.000255 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.000262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.000860 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.001138 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.001148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.002420 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.002426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.003978 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.003988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.005177 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.005186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.005363 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.006061 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.006090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.007287 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.020281 (+ 12994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.020285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.020289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.020770 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.021171 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.021175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.021179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.021310 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.021800 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.021804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.021969 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.022117 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.022429 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":432,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260924 11:41:21.032840 6858 heartbeater.cc:503] Master 127.0.97.62:44733 was elected leader, sending a full tablet report...
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 11:41:21.044530 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.029364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.0.97.61:42233)
0924 11:41:21.029514 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.029519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.030163 (+ 644us) server_negotiation.cc:407] Connection header received
0924 11:41:21.030403 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.030410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.030498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.030610 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.031545 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.032795 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.034838 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.035531 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.035647 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.039603 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.039607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.039610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.039636 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.042133 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.042727 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.042733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.042736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.042847 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.043197 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.043203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.043206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.043429 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.043565 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.043944 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.044146 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:41:21.044571 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.029428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55462)
0924 11:41:21.030028 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.030070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.030211 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.030637 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.030643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.031224 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.031413 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.031421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.032932 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.032938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.034693 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.034710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.035692 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.035701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.035942 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.036625 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.036651 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.039390 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.042260 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.042271 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.042277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.042623 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.042944 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.042949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.042952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.043094 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.043542 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.043549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.043704 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.043876 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.044148 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":434,"threads_started":1}
I20260924 11:41:21.046118 6534 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:42233: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 42233
}
attrs {
promote: true
}
}
dest_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170"
cas_config_opid_index: -1
I20260924 11:41:21.047032 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } }
I20260924 11:41:21.054164 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } }
I20260924 11:41:21.055569 7038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:21.058559 7038 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Refusing update from remote peer 57cdef02ecfa4c998ae8a9e3f5c41170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 11:41:21.059005 7093 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:42233
I20260924 11:41:21.059656 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } }
I20260924 11:41:21.059922 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:21.059958 6996 master.cc:537] Master@127.0.97.61:42233 shutting down...
I20260924 11:41:21.059859 7094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.60.keytab.
W20260924 11:41:21.063374 6520 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 11:41:21.065294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39555
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:41:21.074196 6996 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 11:41:21.074466 6520 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 -> Peer 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61:42233): Couldn't send request to peer 8898367107cd49fb8bd6dccff975f1d3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 11:41:21.074608 6996 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:21.074791 6996 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: stopping tablet replica
I20260924 11:41:21.081118 6996 master.cc:552] Master@127.0.97.61:42233 shutdown complete.
I20260924 11:41:21.081185 6996 master_runner.cc:305] Clearing existing system tablet
I20260924 11:41:21.081519 6996 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:21.081580 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:21.083894 6996 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:21.085423 7099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:21.085983 6996 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:21.086344 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 11:41:21.086455 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
uuid: "8898367107cd49fb8bd6dccff975f1d3"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "04dce6dbe5d98699d1bfef249b482873"
server_key_iv: "05071f2bf458fafa4f1f4ef121aee5a7"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.089866 6996 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:21.094590 6996 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 11:41:21.094753 6996 log.cc:1196] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 11:41:21.095094 6996 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Deleting consensus metadata
I20260924 11:41:21.095325 6996 master_runner.cc:315] Copying system tablet from 127.0.97.62:44733
I20260924 11:41:21.097555 6996 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:44733
I20260924 11:41:21.109508 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.098249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.0.97.62:44733)
0924 11:41:21.098461 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.098469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.098950 (+ 481us) server_negotiation.cc:407] Connection header received
0924 11:41:21.099153 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.099159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.099244 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.099357 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.100068 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.100843 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.102063 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.102453 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.102603 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.104374 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.104381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.104385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.104422 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.107064 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.107649 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.107655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.107659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.107778 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.108165 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.108173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.108177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.108431 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.108613 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.108969 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.109182 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:41:21.109599 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.098191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:43610)
0924 11:41:21.098841 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.098856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.099016 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.099377 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.099381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.099748 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.099931 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.099940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.100972 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.100977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.101915 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.101926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.102614 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.102619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.102765 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.103419 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.103434 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.104133 (+ 699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.107200 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.107205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.107210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.107533 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.107899 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.107905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.107908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.108030 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.108562 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.108566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.108727 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.108885 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.109178 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":409,"threads_started":1}
I20260924 11:41:21.110584 6555 tablet_copy_service.cc:140] P 57cdef02ecfa4c998ae8a9e3f5c41170: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8898367107cd49fb8bd6dccff975f1d3 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43610)
I20260924 11:41:21.110823 6555 tablet_copy_service.cc:161] P 57cdef02ecfa4c998ae8a9e3f5c41170: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8898367107cd49fb8bd6dccff975f1d3 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43610: session id = 8898367107cd49fb8bd6dccff975f1d3-00000000000000000000000000000000
I20260924 11:41:21.113842 6555 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 11:41:21.115449 6996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 11:41:21.119732 6996 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: tablet copy: Starting download of 0 data blocks...
I20260924 11:41:21.120000 6996 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: tablet copy: Starting download of 1 WAL segments...
I20260924 11:41:21.122220 6996 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 11:41:21.125193 6555 tablet_copy_service.cc:342] P 57cdef02ecfa4c998ae8a9e3f5c41170: Request end of tablet copy session 8898367107cd49fb8bd6dccff975f1d3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43610
I20260924 11:41:21.125358 6555 tablet_copy_service.cc:433] P 57cdef02ecfa4c998ae8a9e3f5c41170: ending tablet copy session 8898367107cd49fb8bd6dccff975f1d3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8898367107cd49fb8bd6dccff975f1d3
I20260924 11:41:21.138922 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:21.141273 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:21.147069 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:21.155720 6996 server_base.cc:1034] running on GCE node
W20260924 11:41:21.156024 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:21.156388 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:21.157001 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:21.158190 6996 hybrid_clock.cc:648] HybridClock initialized: now 1790250081158142 us; error 40 us; skew 500 ppm
I20260924 11:41:21.158397 6996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250081, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:21.160326 6996 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 11:41:21.164834 6996 webserver.cc:460] Webserver started at http://127.0.97.61:33445/ using document root <none> and password file <none>
I20260924 11:41:21.165143 6996 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:21.165226 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:21.167383 6996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 11:41:21.168985 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:21.169410 6996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 11:41:21.169544 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
uuid: "8898367107cd49fb8bd6dccff975f1d3"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "04dce6dbe5d98699d1bfef249b482873"
server_key_iv: "05071f2bf458fafa4f1f4ef121aee5a7"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.169700 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:21.178373 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:21.181564 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:21.210470 6996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42233
I20260924 11:41:21.210492 7167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42233 every 8 connection(s)
I20260924 11:41:21.211315 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
I20260924 11:41:21.218062 7168 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:41:21.223068 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Bootstrap starting.
I20260924 11:41:21.234778 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:41:21.235175 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: Bootstrap complete.
I20260924 11:41:21.235901 7168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } }
I20260924 11:41:21.236105 7168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8898367107cd49fb8bd6dccff975f1d3, State: Initialized, Role: LEARNER
I20260924 11:41:21.236344 7168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } }
I20260924 11:41:21.237077 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } }
I20260924 11:41:21.237358 7168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:21.237463 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:41:21.243718 7182 catalog_manager.cc:1259] Loaded cluster ID: e47f00f057ad4e95bb918af8deb82a2b
I20260924 11:41:21.243794 7182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: loading cluster ID for follower catalog manager: success
I20260924 11:41:21.246806 7182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: acquiring CA information for follower catalog manager: success
I20260924 11:41:21.248922 7182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:41:21.426385 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:21.426690 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:21.426851 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:21.434083 7097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:21.434208 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:21.434309 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:21.434394 7097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:21.434469 7097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:21.443787 7097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:21.444883 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:21.446272 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:41:21.455039 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.447140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55466)
0924 11:41:21.447311 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.447334 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.447477 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.448923 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.448928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.448969 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:21.449173 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.449182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.451093 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.451100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.452673 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.452688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.454290 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.454297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.454514 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.454583 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.454653 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.454788 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 11:41:21.456072 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.447267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.97.61:42233)
0924 11:41:21.448594 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.448598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.448617 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:41:21.448670 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.448673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.448749 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.448878 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:21.449319 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.450893 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.452868 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.454100 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.455363 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.455575 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.455623 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.455771 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":1095,"threads_started":1}
I20260924 11:41:21.457163 7143 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 11:41:21.457682 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cdef02ecfa4c998ae8a9e3f5c41170. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } }
I20260924 11:41:21.457952 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: LEARNER
W20260924 11:41:21.461948 7189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:21.462253 7188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:21.462747 7191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:21.463742 7097 server_base.cc:1034] running on GCE node
I20260924 11:41:21.464511 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:21.465695 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:21.466913 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790250081466869 us; error 55 us; skew 500 ppm
I20260924 11:41:21.467233 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250081, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:21.470243 7097 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 11:41:21.471529 7097 webserver.cc:460] Webserver started at http://127.0.97.60:40103/ using document root <none> and password file <none>
I20260924 11:41:21.472235 7097 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:21.472369 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:21.472654 7097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:21.475404 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/instance:
uuid: "4ff056c2d472434aae750f62f7b310c3"
format_stamp: "Formatted at 2026-09-24 11:41:21 on dist-test-slave-qhrh"
server_key: "1f4788bce8bdf74497dd0885673eb557"
server_key_iv: "bf0d8fa9fd62fc1ef3a8061861325c4f"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.476212 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal/instance:
uuid: "4ff056c2d472434aae750f62f7b310c3"
format_stamp: "Formatted at 2026-09-24 11:41:21 on dist-test-slave-qhrh"
server_key: "1f4788bce8bdf74497dd0885673eb557"
server_key_iv: "bf0d8fa9fd62fc1ef3a8061861325c4f"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.479344 6991 heartbeater.cc:503] Master 127.0.97.62:44733 was elected leader, sending a full tablet report...
I20260924 11:41:21.481396 7097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 11:41:21.484558 7198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:21.485785 7097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:41:21.485975 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
uuid: "4ff056c2d472434aae750f62f7b310c3"
format_stamp: "Formatted at 2026-09-24 11:41:21 on dist-test-slave-qhrh"
server_key: "1f4788bce8bdf74497dd0885673eb557"
server_key_iv: "bf0d8fa9fd62fc1ef3a8061861325c4f"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.486204 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:21.500205 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:21.503751 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:21.504014 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:21.522040 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39555
I20260924 11:41:21.522080 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39555 every 8 connection(s)
I20260924 11:41:21.523204 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
I20260924 11:41:21.526345 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:21.529353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 7097
I20260924 11:41:21.533318 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250081, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:21.551995 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.536404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.0.97.62:44733)
0924 11:41:21.536573 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.536577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.537146 (+ 569us) server_negotiation.cc:407] Connection header received
0924 11:41:21.537362 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.537366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.537424 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.537548 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.538807 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.539542 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.541699 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.542102 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.542191 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.545636 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.545640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.545643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.545681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.549076 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.549838 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.549842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.549844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.549992 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.550392 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.550396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.550401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.550622 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.550717 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.551419 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.551621 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:41:21.552960 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.531634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.97.60:39555)
0924 11:41:21.532340 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.532350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.532386 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:41:21.533929 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.533948 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.534280 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.534574 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.536238 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.537966 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.539502 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.540315 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.540435 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.545252 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.545261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.545278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.545314 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.549246 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.549938 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.549944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.549947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.550104 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.550436 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.550441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.550444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.551089 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.551332 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.551599 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.551809 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":539,"thread_start_us":315,"threads_started":1}
I20260924 11:41:21.552978 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.536333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:43622)
0924 11:41:21.537026 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.537063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.537217 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.537574 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.537580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.538344 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.538618 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.538629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.539668 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.539672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.541156 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.541169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.542315 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.542324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.542506 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.543037 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.543073 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.545476 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.549174 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.549184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.549193 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.549690 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.550103 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.550107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.550110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.550269 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.550735 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.550740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.551082 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.551320 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.551728 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":118,"spinlock_wait_cycles":26752,"thread_start_us":357,"threads_started":1}
I20260924 11:41:21.554334 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250081, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 11:41:21.565896 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.555023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55482)
0924 11:41:21.555184 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.555202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.555293 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.555580 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.555583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.555942 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.556116 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.556126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.557065 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.557069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.557907 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.557916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.558592 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.558599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.558718 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.559127 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.559141 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.561233 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.564133 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.564171 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.564176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.564501 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.564795 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.564798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.564800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.564911 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.565345 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.565347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.565440 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.565527 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.565693 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:41:21.565992 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.555101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.97.61:42233)
0924 11:41:21.555275 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.555280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.555298 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:41:21.555383 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.555388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.555457 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.555554 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.556309 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.556957 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.558005 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.558456 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.558569 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.561382 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.561387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.561389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.561417 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.564024 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.564594 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.564597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.564599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.564708 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.564992 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.564995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.564997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.565237 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.565372 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.565612 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.565758 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:41:21.566866 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250081, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 11:41:21.578378 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.567474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.0.1:38390)
0924 11:41:21.567592 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.567605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.567701 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.567954 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.567958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.568339 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.568453 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.568460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.569345 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.569353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.570153 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.570162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.570821 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.570826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.571063 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.571439 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.571451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.573192 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.576189 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.576193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.576196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.576561 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.576892 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.576896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.576900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.577029 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.577468 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.577473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.577605 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.577724 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.577972 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":20}
I20260924 11:41:21.578385 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.567606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.97.60:39555)
0924 11:41:21.567728 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.567736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.567753 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:41:21.567793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.567796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.567853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.567941 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.568598 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.569221 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.570274 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.570681 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.570815 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.573348 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.573353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.573358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.573390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.576064 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.576666 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.576674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.576677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.576785 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.577133 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.577138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.577142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.577359 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.577488 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.577810 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.578093 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":44}
I20260924 11:41:21.583150 7250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Bootstrap starting.
I20260924 11:41:21.585836 7250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:21.586979 7250 log.cc:824] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:21.589432 7250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: No bootstrap required, opened a new log
I20260924 11:41:21.596242 7250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } }
I20260924 11:41:21.596518 7250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:21.596647 7250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff056c2d472434aae750f62f7b310c3, State: Initialized, Role: FOLLOWER
I20260924 11:41:21.597185 7250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } }
I20260924 11:41:21.597929 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } } }
I20260924 11:41:21.598250 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:41:21.598740 7250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:41:21.607492 7267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:41:21.607666 7267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:41:21.608011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6596
I20260924 11:41:21.612385 7097 master_runner.cc:428] Detected that this master 4ff056c2d472434aae750f62f7b310c3 is joining an existing cluster
I20260924 11:41:21.612504 7097 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:39555
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 11:41:21.635546 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.609032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.97.1:39837)
0924 11:41:21.611275 (+ 2243us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.611281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.611301 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:41:21.611349 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.611353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.611510 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.611681 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.613298 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.614850 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.617577 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.618695 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.625737 (+ 7042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.626053 (+ 316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.626064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.626083 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.626124 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.630896 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.631901 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.631912 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.631918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.632076 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.632618 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.632625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.632631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.633096 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.634760 (+ 1664us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.634908 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.635141 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2099,"thread_start_us":1931,"threads_started":1}
I20260924 11:41:21.637040 6719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:21.658030 7256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:21.658295 7256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } }
I20260924 11:41:21.658919 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.616914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.0.97.61:42233)
0924 11:41:21.617096 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.617105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.641080 (+ 23975us) server_negotiation.cc:407] Connection header received
0924 11:41:21.641977 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.641985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.642109 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.642191 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.644039 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.645402 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.648939 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.650033 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.650190 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.652465 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.652470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.652474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.652508 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.655811 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.656708 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.656715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.656720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.656856 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.657305 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.657310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.657315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.657631 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.657793 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.658396 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.658598 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:41:21.659819 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.616924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55488)
0924 11:41:21.640985 (+ 24061us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.641003 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.641705 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.642461 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.642468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.643289 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.643751 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.643782 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.645652 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.645660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.648778 (+ 3118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.648797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.650463 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.650469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.650665 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.651245 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.651268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.652288 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.655991 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.655996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.656001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.656569 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.656997 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.657003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.657007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.657174 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.657856 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.657881 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.658105 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.659092 (+ 987us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.659359 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23933,"mutex_wait_us":54,"thread_start_us":15227,"threads_started":1}
I20260924 11:41:21.659790 7256 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733), 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61:42233)
I20260924 11:41:21.660650 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.616051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.0.97.62:44733)
0924 11:41:21.616228 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.616234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.639298 (+ 23064us) server_negotiation.cc:407] Connection header received
0924 11:41:21.640396 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.640403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.640481 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.640562 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.641847 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.642903 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.644898 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.645490 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.645579 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.648135 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.648177 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.648181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.648219 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.651901 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.652948 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.652953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.652959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.653102 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.658646 (+ 5544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.658651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.658654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.658990 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.659126 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.660121 (+ 995us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.660367 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:41:21.661244 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff056c2d472434aae750f62f7b310c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8898367107cd49fb8bd6dccff975f1d3" is_pre_election: true
I20260924 11:41:21.661425 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.616517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:43626)
0924 11:41:21.639171 (+ 22654us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.639201 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.640241 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.640716 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.640723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.641379 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.641675 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.641686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.643095 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.643100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.644740 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.644753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.645871 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.645876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.646055 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.646699 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.646735 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.647927 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.652287 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.652293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.652296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.652751 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.653432 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.653436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.653440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.653623 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.659327 (+ 5704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.659334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.659508 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.660849 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.661139 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18725,"mutex_wait_us":2,"thread_start_us":22940,"threads_started":1}
I20260924 11:41:21.662791 6545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff056c2d472434aae750f62f7b310c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" is_pre_election: true
I20260924 11:41:21.663628 7202 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ff056c2d472434aae750f62f7b310c3; no voters: 57cdef02ecfa4c998ae8a9e3f5c41170, 8898367107cd49fb8bd6dccff975f1d3
I20260924 11:41:21.664248 7256 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 11:41:21.666561 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.616200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.0.97.60:39555)
0924 11:41:21.616354 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.616361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.632639 (+ 16278us) server_negotiation.cc:407] Connection header received
0924 11:41:21.632866 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.632873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.632947 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.633017 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.635098 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.636790 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.639271 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.640399 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.640503 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.653991 (+ 13488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.653997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.654002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.654038 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.657637 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.661635 (+ 3998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.661640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.661644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.661779 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.664819 (+ 3040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.664823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.664829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.665189 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.665288 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.665999 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.666200 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 11:41:21.669996 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.616296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.0.1:38406)
0924 11:41:21.632536 (+ 16240us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.632560 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.632738 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.633227 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.633234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.634031 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.634466 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.634481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.637056 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.637063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.639105 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.639122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.645838 (+ 6716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.645847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.646751 (+ 904us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.647400 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.647421 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.653800 (+ 6379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.658423 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.658427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.658431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.659918 (+ 1487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.662425 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.662430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.662433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.664656 (+ 2223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.665411 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.665418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.665596 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.665876 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.669586 (+ 3710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15779,"mutex_wait_us":644,"thread_start_us":18447,"threads_started":1}
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 11:41:21.682901 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.666399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.0.1:38408)
0924 11:41:21.666586 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.666604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.666690 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.667478 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.667483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.667903 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.668077 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.668087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.669407 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.669413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.670836 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.670851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.671942 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.671949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.672166 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.672721 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.672743 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.675410 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.679817 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.679823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.679826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.680276 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.680682 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.680687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.680690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.680902 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.681693 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.681698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.681886 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.682096 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.682386 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 11:41:21.684350 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.666838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.0.97.60:39555)
0924 11:41:21.667036 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.667045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.667076 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:41:21.667156 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.667162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.667277 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.667430 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.668233 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.669226 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.671005 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.671779 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.671912 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.675652 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.675659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.675666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.675727 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.679677 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.680378 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.680383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.680386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.680494 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.681079 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.681083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.681086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.681500 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.683338 (+ 1838us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.683507 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.683826 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":333,"threads_started":1}
I20260924 11:41:21.685698 6534 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:39555: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4ff056c2d472434aae750f62f7b310c3"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 39555
}
attrs {
promote: true
}
}
dest_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170"
cas_config_opid_index: 5
I20260924 11:41:21.686587 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } }
I20260924 11:41:21.693145 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.696733 7094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:39555
I20260924 11:41:21.697445 7097 master.cc:537] Master@127.0.97.60:39555 shutting down...
I20260924 11:41:21.697743 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 LEARNER]: Refusing update from remote peer 57cdef02ecfa4c998ae8a9e3f5c41170: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 11:41:21.698763 7281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 11:41:21.712316 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.712641 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:21.712571 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.716696 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: true } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.716958 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:41:21.717819 7279 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: attempting to promote NON_VOTER 8898367107cd49fb8bd6dccff975f1d3 to VOTER
I20260924 11:41:21.718502 7279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } }
I20260924 11:41:21.720515 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 LEARNER]: Refusing update from remote peer 57cdef02ecfa4c998ae8a9e3f5c41170: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 11:41:21.720988 7280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 11:41:21.722589 6521 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 -> Peer 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60:39555): Couldn't send request to peer 4ff056c2d472434aae750f62f7b310c3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 11:41:21.723188 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.723834 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.726756 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.726967 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:21.727224 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.727427 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:21.727808 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.728030 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:41:21.734210 7097 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:21.734573 7097 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:21.734741 7097 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: stopping tablet replica
I20260924 11:41:21.740875 7097 master.cc:552] Master@127.0.97.60:39555 shutdown complete.
I20260924 11:41:21.741007 7097 master_runner.cc:305] Clearing existing system tablet
I20260924 11:41:21.741437 7097 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:21.741575 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:21.744201 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 11:41:21.745679 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:21.746004 7097 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:21.746363 7097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 11:41:21.746497 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
uuid: "4ff056c2d472434aae750f62f7b310c3"
format_stamp: "Formatted at 2026-09-24 11:41:21 on dist-test-slave-qhrh"
server_key: "1f4788bce8bdf74497dd0885673eb557"
server_key_iv: "bf0d8fa9fd62fc1ef3a8061861325c4f"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.751739 7097 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:21.758224 7097 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 11:41:21.758430 7097 log.cc:1196] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 11:41:21.759409 7097 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Deleting consensus metadata
I20260924 11:41:21.759680 7097 master_runner.cc:315] Copying system tablet from 127.0.97.62:44733
I20260924 11:41:21.762005 7097 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:44733
I20260924 11:41:21.781994 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.762735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:43640)
0924 11:41:21.768530 (+ 5795us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:21.768553 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:21.768710 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:21.769065 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:21.769070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:21.769591 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:21.769862 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.769874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.771344 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.771348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.772882 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:21.772899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.773758 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.773764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:21.773914 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.774579 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:21.774600 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:21.775682 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:21.779458 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.779463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.779467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.779900 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.780392 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:21.780396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:21.780400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:21.780548 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:21.781056 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:21.781061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:21.781219 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 11:41:21.781357 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 11:41:21.781617 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5655,"thread_start_us":5539,"threads_started":1}
I20260924 11:41:21.781991 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:21.762765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.0.97.62:44733)
0924 11:41:21.762978 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:21.762984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:21.768646 (+ 5662us) server_negotiation.cc:407] Connection header received
0924 11:41:21.768844 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:21.768850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:21.768923 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:21.769012 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:21.770031 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.771150 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.773024 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:21.773614 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:21.773725 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:21.775854 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:21.775860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:21.775864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:21.775917 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:21.779329 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.780101 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.780106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.780110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.780283 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:21.780651 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:21.780657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:21.780661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:21.780935 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:21.781083 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:21.781453 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 11:41:21.781677 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:41:21.782896 6555 tablet_copy_service.cc:140] P 57cdef02ecfa4c998ae8a9e3f5c41170: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4ff056c2d472434aae750f62f7b310c3 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43640)
I20260924 11:41:21.783051 6555 tablet_copy_service.cc:161] P 57cdef02ecfa4c998ae8a9e3f5c41170: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4ff056c2d472434aae750f62f7b310c3 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43640: session id = 4ff056c2d472434aae750f62f7b310c3-00000000000000000000000000000000
I20260924 11:41:21.786592 6555 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 11:41:21.788282 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 11:41:21.793619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6596
I20260924 11:41:21.797251 7097 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: tablet copy: Starting download of 0 data blocks...
I20260924 11:41:21.797508 7097 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: tablet copy: Starting download of 1 WAL segments...
I20260924 11:41:21.799431 7097 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 11:41:21.802848 6555 tablet_copy_service.cc:342] P 57cdef02ecfa4c998ae8a9e3f5c41170: Request end of tablet copy session 4ff056c2d472434aae750f62f7b310c3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43640
I20260924 11:41:21.802964 6555 tablet_copy_service.cc:433] P 57cdef02ecfa4c998ae8a9e3f5c41170: ending tablet copy session 4ff056c2d472434aae750f62f7b310c3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4ff056c2d472434aae750f62f7b310c3
I20260924 11:41:21.811434 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:21.813833 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:21.815999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39837
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44733
--webserver_port=41289
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:21.820354 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:21.825453 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:21.825610 7097 server_base.cc:1034] running on GCE node
I20260924 11:41:21.826125 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:21.826727 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:21.827900 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790250081827866 us; error 29 us; skew 500 ppm
I20260924 11:41:21.828109 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:21 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250081, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:21.829830 7097 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 11:41:21.830955 7097 webserver.cc:460] Webserver started at http://127.0.97.60:43835/ using document root <none> and password file <none>
I20260924 11:41:21.831228 7097 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:21.831331 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:21.833362 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 11:41:21.845064 7304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:21.845432 7097 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.004s
I20260924 11:41:21.845542 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
uuid: "4ff056c2d472434aae750f62f7b310c3"
format_stamp: "Formatted at 2026-09-24 11:41:21 on dist-test-slave-qhrh"
server_key: "1f4788bce8bdf74497dd0885673eb557"
server_key_iv: "bf0d8fa9fd62fc1ef3a8061861325c4f"
server_key_version: "encryptionkey@0"
I20260924 11:41:21.845722 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:21.880276 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:21.883476 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:21.902387 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39555
I20260924 11:41:21.902427 7355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39555 every 8 connection(s)
I20260924 11:41:21.903203 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
I20260924 11:41:21.909997 7356 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:41:21.912464 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Bootstrap starting.
I20260924 11:41:21.925040 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:41:21.925437 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Bootstrap complete.
I20260924 11:41:21.926254 7356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } }
I20260924 11:41:21.926501 7356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ff056c2d472434aae750f62f7b310c3, State: Initialized, Role: LEARNER
I20260924 11:41:21.926764 7356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } }
I20260924 11:41:21.927493 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:21.927793 7356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:21.927914 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:41:21.937639 7370 catalog_manager.cc:1259] Loaded cluster ID: e47f00f057ad4e95bb918af8deb82a2b
I20260924 11:41:21.937739 7370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: loading cluster ID for follower catalog manager: success
I20260924 11:41:21.940317 7370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: acquiring CA information for follower catalog manager: success
I20260924 11:41:21.941848 7370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:41:22.205619 7298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:22.205911 7298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:22.206077 7298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:22.214632 7298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:22.214735 7298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:22.214890 7298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:22.227334 7298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41289
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:22.228637 7298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:22.229872 7298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:22.241921 7376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:22.241942 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:22.241885 7375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:22.242681 7298 server_base.cc:1034] running on GCE node
I20260924 11:41:22.243158 7298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:22.243924 7298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:22.245129 7298 hybrid_clock.cc:648] HybridClock initialized: now 1790250082245068 us; error 73 us; skew 500 ppm
I20260924 11:41:22.245389 7298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:22.258692 7298 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 11:41:22.260064 7298 webserver.cc:460] Webserver started at http://127.0.97.1:41289/ using document root <none> and password file <none>
I20260924 11:41:22.260694 7298 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:22.260839 7298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:22.265350 7298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 11:41:22.267699 7385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:22.268890 7298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:22.269043 7298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "834abac14e68482495726b4d94f6f5d5"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "252a2c69058e90ef1ea2b76c98f49d6f"
server_key_iv: "339428389b15b3aad945517964652070"
server_key_version: "encryptionkey@0"
I20260924 11:41:22.269902 7298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:22.283910 7298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:22.287644 7298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:22.287981 7298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:22.288795 7298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:22.290073 7298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:22.290180 7298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:22.290294 7298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:22.290386 7298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:22.310878 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.303287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.0.1:38410)
0924 11:41:22.304344 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.304367 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.304503 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.305309 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.305314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.305341 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:22.305591 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.305607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.307839 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.307845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.309023 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.309033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.310101 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.310106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.310273 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.310327 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.310422 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.310548 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":873,"threads_started":1}
I20260924 11:41:22.310914 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.303434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.0.97.60:39555)
0924 11:41:22.305025 (+ 1591us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.305030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.305045 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:41:22.305102 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.305105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.305166 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.305287 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:22.305746 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.307700 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.309167 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.309913 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.310056 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.310262 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.310457 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.310584 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472,"thread_start_us":1282,"threads_started":1}
I20260924 11:41:22.312079 7331 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 11:41:22.312674 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cdef02ecfa4c998ae8a9e3f5c41170. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:22.312951 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:41:22.330380 7298 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39837
I20260924 11:41:22.330399 7500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39837 every 8 connection(s)
I20260924 11:41:22.331811 7298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:22.332388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 7298
I20260924 11:41:22.332675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6730
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 11:41:22.384344 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.334666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40571 (local address 127.0.97.60:39555)
0924 11:41:22.335100 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.335109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.353975 (+ 18866us) server_negotiation.cc:407] Connection header received
0924 11:41:22.354156 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.354166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.354274 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.354406 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.357404 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.358833 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.362203 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.363118 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.363245 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.369209 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.369218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.369226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.369269 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.373625 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.374692 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.374700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.374705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.374870 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.375377 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.375384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.375387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.375732 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.375890 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.377388 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.377634 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:22.387696 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.335402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.97.1:40571)
0924 11:41:22.353854 (+ 18452us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.353871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.354030 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.354914 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.354929 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.356068 (+ 1139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:22.357134 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.357166 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.359132 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.359140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.361978 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.362000 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.363681 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.363691 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.364074 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.365023 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:22.365064 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:22.368989 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:22.373802 (+ 4813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.373811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.373830 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.374556 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.375019 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.375022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.375025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.375264 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.375906 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:22.375916 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:22.376388 (+ 472us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.384652 (+ 8264us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.386197 (+ 1545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18333,"thread_start_us":12112,"threads_started":1}
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 11:41:22.390434 7501 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39555
I20260924 11:41:22.390746 7501 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:22.391635 7501 heartbeater.cc:511] Master 127.0.97.60:39555 requested a full tablet report, sending...
I20260924 11:41:22.393162 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.335111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:49313)
0924 11:41:22.348705 (+ 13594us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.348763 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.352175 (+ 3412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.369044 (+ 16869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.369051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.369542 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:22.369845 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.369860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.372958 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.372967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.374642 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.374655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.377871 (+ 3216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.377879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.378187 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.378879 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:22.378915 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:22.385513 (+ 6598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:22.389921 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.389926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.389932 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.390258 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.391310 (+ 1052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.391314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.391318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.391467 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.392249 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:22.392257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:22.392401 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.392532 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.392771 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12273,"thread_start_us":41273,"threads_started":1}
I20260924 11:41:22.393465 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.335692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49313 (local address 127.0.97.62:44733)
0924 11:41:22.368670 (+ 32978us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.368679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.368704 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:41:22.368774 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.368781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.368889 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.369023 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.371209 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.372818 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.376673 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.377569 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.377722 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.385687 (+ 7965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.385693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.385698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.385736 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.389593 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.390394 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.390400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.390406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.390558 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.391764 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.391769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.391771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.392100 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.392774 (+ 674us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.392895 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.393132 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32810,"thread_start_us":33492,"threads_started":1}
I20260924 11:41:22.394048 7321 ts_manager.cc:194] Registered new tserver with Master: 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837)
I20260924 11:41:22.394264 7502 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44733
I20260924 11:41:22.394397 7502 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:22.394855 7502 heartbeater.cc:511] Master 127.0.97.62:44733 requested a full tablet report, sending...
I20260924 11:41:22.395992 6534 ts_manager.cc:194] Re-registered known tserver with Master: 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837)
I20260924 11:41:22.396116 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.339804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.97.1:57799)
0924 11:41:22.379955 (+ 40151us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.379970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.380080 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.380479 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.380483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.380895 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:22.381141 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.381149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.382570 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.382575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.383721 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.383733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.384833 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.384841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.385016 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.385609 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:22.385627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:22.388821 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:22.392515 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.392519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.392522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.393704 (+ 1182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.394284 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.394288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.394292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.394464 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.395219 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:22.395226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:22.395369 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.395526 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.395746 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37549,"thread_start_us":13795,"threads_started":1}
I20260924 11:41:22.396246 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.340450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57799 (local address 127.0.97.61:42233)
0924 11:41:22.353435 (+ 12985us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.353444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.380065 (+ 26621us) server_negotiation.cc:407] Connection header received
0924 11:41:22.380196 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.380201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.380328 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.380440 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.381267 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.382426 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.383838 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.384457 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.384600 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.389019 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.389026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.389029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.389063 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.392341 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.393861 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.393868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.393871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.394013 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.394630 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.394638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.394644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.394973 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.395098 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.395626 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.395834 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12837,"thread_start_us":12701,"threads_started":1}
I20260924 11:41:22.397109 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49313
I20260924 11:41:22.397446 7503 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42233
I20260924 11:41:22.397599 7503 heartbeater.cc:464] Registering TS with master...
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 11:41:22.398509 7503 heartbeater.cc:511] Master 127.0.97.61:42233 requested a full tablet report, sending...
I20260924 11:41:22.400089 7133 ts_manager.cc:194] Registered new tserver with Master: 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837)
I20260924 11:41:22.404285 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.333860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.0.97.2:37021)
0924 11:41:22.376328 (+ 42468us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.376340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.378559 (+ 2219us) server_negotiation.cc:407] Connection header received
0924 11:41:22.385014 (+ 6455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.385022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.385266 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.385416 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.388170 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.389501 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.391242 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.392043 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.392184 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.398603 (+ 6419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.398616 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.398630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.398667 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.401518 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.402162 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.402170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.402174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.402300 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.402668 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.402673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.402677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.402973 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.403269 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.403558 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.403817 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42224,"thread_start_us":42688,"threads_started":1}
I20260924 11:41:22.405411 6853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:22.485810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6730
I20260924 11:41:22.501684 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37021
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44733
--webserver_port=40879
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:22.802095 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:22.802315 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:22.802398 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:22.809481 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:22.809566 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:22.809717 7513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:22.818678 7513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40879
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:22.819784 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:22.821017 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:41:22.821931 7426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: true } } }
I20260924 11:41:22.822316 7426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:22.823423 7518 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: attempting to promote NON_VOTER 4ff056c2d472434aae750f62f7b310c3 to VOTER
I20260924 11:41:22.824059 7518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } }
I20260924 11:41:22.826318 7331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 LEARNER]: Refusing update from remote peer 57cdef02ecfa4c998ae8a9e3f5c41170: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 11:41:22.826388 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Refusing update from remote peer 57cdef02ecfa4c998ae8a9e3f5c41170: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 11:41:22.826898 7426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 11:41:22.827244 7518 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 11:41:22.830226 7518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.832267 7331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.832653 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.835282 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.835585 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:22.835827 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.836038 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:22.836251 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.836467 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:22.837232 7526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.837599 7526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:41:22.838142 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:22.838527 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 11:41:22.845638 7522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:22.845665 7529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:22.845750 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:22.849488 7513 server_base.cc:1034] running on GCE node
I20260924 11:41:22.850088 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:22.850929 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:22.852177 7513 hybrid_clock.cc:648] HybridClock initialized: now 1790250082852097 us; error 41 us; skew 500 ppm
I20260924 11:41:22.852451 7513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:22.855227 7513 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 11:41:22.856645 7513 webserver.cc:460] Webserver started at http://127.0.97.2:40879/ using document root <none> and password file <none>
I20260924 11:41:22.857252 7513 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:22.857352 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:22.862351 7513 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:41:22.864912 7536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:22.866111 7513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:41:22.866227 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "99913db5b7e9468daf0f29cbb0971b4a"
format_stamp: "Formatted at 2026-09-24 11:41:19 on dist-test-slave-qhrh"
server_key: "ea037925ffe6f38b545a56dd73485644"
server_key_iv: "c0a72a50060d5a8f475dfa5b283f9090"
server_key_version: "encryptionkey@0"
I20260924 11:41:22.867095 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:22.887346 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:22.890771 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:22.890961 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:22.891712 7513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:22.892912 7513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:22.892985 7513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:22.893047 7513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:22.893095 7513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:22.927366 7513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37021
I20260924 11:41:22.927503 7648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37021 every 8 connection(s)
I20260924 11:41:22.928777 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:22.939031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 7513
I20260924 11:41:22.939311 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6863
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250082, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 11:41:22 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 11:41:22.971743 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.932466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37271 (local address 127.0.97.60:39555)
0924 11:41:22.932992 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.933000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.944982 (+ 11982us) server_negotiation.cc:407] Connection header received
0924 11:41:22.948308 (+ 3326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.948316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.948427 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.948513 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.951012 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.952491 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.957043 (+ 4552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.957775 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.957876 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.963695 (+ 5819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.963701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.963706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.963746 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.967899 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.969045 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.969051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.969055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.969181 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.969672 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.969678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.969681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.969992 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.970134 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.971074 (+ 940us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.971270 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":392,"threads_started":1}
I20260924 11:41:22.973625 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.932012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.97.2:37271)
0924 11:41:22.944058 (+ 12046us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.944131 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.947971 (+ 3840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.949081 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.949092 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.949908 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:22.950765 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.950791 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.953106 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.953130 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.956878 (+ 3748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.956894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.958368 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.958380 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.958902 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.960498 (+ 1596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:22.960523 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:22.963438 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:22.968300 (+ 4862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.968311 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.968328 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.968919 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.969318 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.969323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.969327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.969543 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.970228 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:22.970237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:22.970670 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.971507 (+ 837us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.972275 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11708,"thread_start_us":11953,"threads_started":1}
I20260924 11:41:22.975369 7649 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39555
I20260924 11:41:22.975656 7649 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:22.976502 7649 heartbeater.cc:511] Master 127.0.97.60:39555 requested a full tablet report, sending...
I20260924 11:41:22.977351 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.933115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.97.2:43607)
0924 11:41:22.955862 (+ 22747us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.955884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.955996 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.959014 (+ 3018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.959020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.960114 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:22.960448 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.960463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.962124 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.962129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.963606 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.963619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.964915 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.964922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.965076 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.965593 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:22.965611 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:22.968569 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:22.974042 (+ 5473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.974049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.974055 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.975162 (+ 1107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.975551 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.975556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.975560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.975695 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.976340 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:22.976346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:22.976545 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.976711 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.976953 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22603,"mutex_wait_us":18,"thread_start_us":22168,"threads_started":1}
I20260924 11:41:22.977756 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.931611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43607 (local address 127.0.97.61:42233)
0924 11:41:22.958697 (+ 27086us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.958704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.958721 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:41:22.958775 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.958779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.958845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.960610 (+ 1765us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.960785 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.961859 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.963771 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.964709 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.965353 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.968778 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.968786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.968791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.968897 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.973901 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.975292 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.975297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.975300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.975426 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.975840 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.975845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.975850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.976211 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.976515 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.976821 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.977034 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26971,"thread_start_us":30331,"threads_started":1}
I20260924 11:41:22.978143 7321 ts_manager.cc:194] Registered new tserver with Master: 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021)
I20260924 11:41:22.978428 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.932785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53269 (local address 127.0.97.62:44733)
0924 11:41:22.933312 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.933319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.960090 (+ 26771us) server_negotiation.cc:407] Connection header received
0924 11:41:22.960323 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.960329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.960415 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.960496 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.961715 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.962793 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.965108 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.965789 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.965880 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.969638 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.969647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.969654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.969711 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.973579 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.975048 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.975053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.975056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.975190 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.975760 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.975768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.975771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.976120 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.976264 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.977892 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.978084 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":545,"threads_started":1}
I20260924 11:41:22.978631 7651 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42233
I20260924 11:41:22.978749 7651 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:22.979205 7651 heartbeater.cc:511] Master 127.0.97.61:42233 requested a full tablet report, sending...
I20260924 11:41:22.979244 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.932012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:53269)
0924 11:41:22.959975 (+ 27963us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:22.959988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:22.960157 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:22.960709 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:22.960713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:22.961259 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:22.961550 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.961561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.962956 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.962960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.964927 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:22.964942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.966146 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.966155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:22.966337 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.966897 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:22.966922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:22.969433 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:22.973766 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.973771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.973774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.974922 (+ 1148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.975390 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:22.975395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:22.975398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:22.975544 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:22.976342 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:22.976347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:22.977620 (+ 1273us) client_negotiation.cc:769] Sending connection context
0924 11:41:22.978640 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0924 11:41:22.978880 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27858,"mutex_wait_us":1447,"thread_start_us":26721,"threads_started":1}
I20260924 11:41:22.979784 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:22.940363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.0.97.3:41155)
0924 11:41:22.955973 (+ 15610us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:22.955984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:22.958151 (+ 2167us) server_negotiation.cc:407] Connection header received
0924 11:41:22.958231 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:22.958238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:22.958445 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:22.958627 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:22.961143 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.962518 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.964220 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:22.965271 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:22.965351 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:22.972316 (+ 6965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:22.972328 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:22.972342 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:22.972402 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:22.976037 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.976848 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.976855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.976859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.976996 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:22.977411 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:22.977418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:22.977422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:22.977812 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:22.978094 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:22.979112 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0924 11:41:22.979337 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15422,"thread_start_us":15113,"threads_started":1}
I20260924 11:41:22.980005 7650 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44733
I20260924 11:41:22.980130 7650 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:22.980512 7650 heartbeater.cc:511] Master 127.0.97.62:44733 requested a full tablet report, sending...
I20260924 11:41:22.980813 7133 ts_manager.cc:194] Registered new tserver with Master: 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021)
I20260924 11:41:22.980894 6986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:22.981362 6534 ts_manager.cc:194] Re-registered known tserver with Master: 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021)
I20260924 11:41:22.983119 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53269
I20260924 11:41:23.065289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6863
I20260924 11:41:23.080648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41155
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44733
--webserver_port=42377
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:23.385581 7662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:23.385788 7662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:23.385864 7662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:23.392799 7662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:23.392899 7662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:23.393042 7662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:23.400151 7502 heartbeater.cc:503] Master 127.0.97.62:44733 was elected leader, sending a full tablet report...
I20260924 11:41:23.402741 7662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42377
--tserver_master_addrs=127.0.97.62:44733,127.0.97.61:42233,127.0.97.60:39555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:23.403766 7662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:23.404865 7662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:23.416843 7668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:23.416807 7669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:23.416788 7671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:23.417822 7662 server_base.cc:1034] running on GCE node
I20260924 11:41:23.418298 7662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:23.419046 7662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:23.420225 7662 hybrid_clock.cc:648] HybridClock initialized: now 1790250083420168 us; error 46 us; skew 500 ppm
I20260924 11:41:23.420522 7662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:23 dist-test-slave-qhrh krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250083, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:23.423607 7662 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 11:41:23.425024 7662 webserver.cc:460] Webserver started at http://127.0.97.3:42377/ using document root <none> and password file <none>
I20260924 11:41:23.425676 7662 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:23.425835 7662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:23.431228 7662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:41:23.434187 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:23.435422 7662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:41:23.435616 7662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "5d468aa0894d49cf8c3ebb1997aaa21f"
format_stamp: "Formatted at 2026-09-24 11:41:20 on dist-test-slave-qhrh"
server_key: "9cc5d87ba37322e583966a59b1822cdb"
server_key_iv: "64485fc036ea9eb90d828d483e8fa8c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:23.436561 7662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:23.449040 7662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:23.452771 7662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:23.453006 7662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:23.453766 7662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:23.454890 7662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:23.455003 7662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:23.455106 7662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:23.455196 7662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:23.485903 7662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41155
I20260924 11:41:23.485976 7790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41155 every 8 connection(s)
I20260924 11:41:23.487223 7662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:23.487708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 7662
Sep 24 11:41:23 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250083, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 11:41:23 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250083, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 11:41:23.545147 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.490016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38243 (local address 127.0.97.62:44733)
0924 11:41:23.505286 (+ 15270us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.505292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.519005 (+ 13713us) server_negotiation.cc:407] Connection header received
0924 11:41:23.519170 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.519177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.519272 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.519383 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.522141 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.523447 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.530680 (+ 7233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.531331 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.531446 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.536545 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.536552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.536557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.536606 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.540921 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.541988 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.541992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.541995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.542133 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.543244 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.543249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.543252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.543534 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.543653 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.544561 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.544745 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15144,"thread_start_us":15279,"threads_started":1}
Sep 24 11:41:23 dist-test-slave-qhrh krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250083, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 11:41:23.547648 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.490636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:38243)
0924 11:41:23.518906 (+ 28270us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.518927 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.519056 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.520173 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.520187 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.521083 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:23.521969 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.521993 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.528171 (+ 6178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.528179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.530538 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.530553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.532490 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.532500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.532759 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.533573 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:23.533602 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:23.536319 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:23.541086 (+ 4767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:23.541093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:23.541106 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:23.541603 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:23.542310 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:23.542315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:23.542318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:23.542508 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:23.543735 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:23.543742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:23.544125 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.545677 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.546219 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28095,"mutex_wait_us":330,"thread_start_us":28117,"threads_started":1}
I20260924 11:41:23.550256 7792 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44733
I20260924 11:41:23.550477 7792 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:23.551245 7792 heartbeater.cc:511] Master 127.0.97.62:44733 requested a full tablet report, sending...
I20260924 11:41:23.552778 6534 ts_manager.cc:194] Re-registered known tserver with Master: 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155)
I20260924 11:41:23.553834 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38243
I20260924 11:41:23.554420 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.490636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.97.3:54929)
0924 11:41:23.503143 (+ 12507us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.503190 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.505937 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.524896 (+ 18959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.524903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.525412 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:23.525703 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.525716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.532358 (+ 6642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.532364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.533650 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.533663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.535604 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.535611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.535811 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.536379 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:23.536404 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:23.539418 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:23.549326 (+ 9908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:23.549347 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:23.549350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:23.549748 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:23.552637 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:23.552646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:23.552651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:23.552815 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:23.553497 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:23.553501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:23.553628 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.553751 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.554028 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12160,"thread_start_us":37834,"threads_started":1}
I20260924 11:41:23.555929 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.490143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54929 (local address 127.0.97.60:39555)
0924 11:41:23.522451 (+ 32308us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.522459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.522483 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:41:23.522549 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.522556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.522668 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.525888 (+ 3220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.526097 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.532252 (+ 6155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.533829 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.534420 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.536089 (+ 1669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.539680 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.539686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.539690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.539720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.544121 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.552377 (+ 8256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.552381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.552385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.552516 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.552931 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.552936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.552939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.553332 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.555321 (+ 1989us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.555443 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.555617 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":32162,"thread_start_us":32089,"threads_started":1}
I20260924 11:41:23.557629 7791 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39555
I20260924 11:41:23.557772 7791 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:23.558228 7791 heartbeater.cc:511] Master 127.0.97.60:39555 requested a full tablet report, sending...
I20260924 11:41:23.560045 7321 ts_manager.cc:194] Registered new tserver with Master: 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155)
I20260924 11:41:23.562217 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.491125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.97.3:52695)
0924 11:41:23.519671 (+ 28546us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.519689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.519793 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.523590 (+ 3797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.523598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.524298 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:23.524678 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.524693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.531356 (+ 6663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.531362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.534810 (+ 3448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.534823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.542322 (+ 7499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.542328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.542503 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.544318 (+ 1815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:23.544335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:23.546499 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:23.557662 (+ 11163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:23.557665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:23.557667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:23.558212 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:23.558881 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:23.558886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:23.558889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:23.559051 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:23.561207 (+ 2156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:23.561212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:23.561375 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.561509 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.561807 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28409,"thread_start_us":15161,"threads_started":1}
I20260924 11:41:23.563956 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.490354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52695 (local address 127.0.97.61:42233)
0924 11:41:23.523079 (+ 32725us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.523084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.523098 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:41:23.523150 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.523157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.523240 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.523377 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.527620 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.531208 (+ 3588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.540091 (+ 8883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.542000 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.542102 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.546696 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.546702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.546706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.546795 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.557527 (+ 10732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.558367 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.558374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.558382 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.558531 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.559896 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.559902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.559905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.561004 (+ 1099us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.561129 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.563400 (+ 2271us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.563612 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32632,"thread_start_us":32538,"threads_started":1}
I20260924 11:41:23.564709 7793 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42233
I20260924 11:41:23.564821 7793 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:23.565104 7793 heartbeater.cc:511] Master 127.0.97.61:42233 requested a full tablet report, sending...
I20260924 11:41:23.566174 7133 ts_manager.cc:194] Registered new tserver with Master: 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155)
I20260924 11:41:23.566519 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.514856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.0.97.60:39555)
0924 11:41:23.544416 (+ 29560us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.544423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.544444 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:23.544496 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.544503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.544584 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.544696 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.548940 (+ 4244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.550103 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.552532 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.553105 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.553187 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.559098 (+ 5911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.559105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.559112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.559148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.562886 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.563636 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.563643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.563647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.563801 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.564384 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.564390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.564393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.564723 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.564954 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.565952 (+ 998us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.566154 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":29413,"thread_start_us":29999,"threads_started":1}
I20260924 11:41:23.568313 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.523064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.0.97.61:42233)
0924 11:41:23.529477 (+ 6413us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.529482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.542240 (+ 12758us) server_negotiation.cc:407] Connection header received
0924 11:41:23.542645 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.542650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.542936 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.543036 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.551288 (+ 8252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.552428 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.556283 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.556893 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.557000 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.561868 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.561875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.561879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.561909 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.565172 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.566023 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.566028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.566033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.566203 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.566627 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.566634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.566639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.566986 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.567222 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.567750 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.567935 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6303,"thread_start_us":6417,"threads_started":1}
I20260924 11:41:23.571908 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.511192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.97.62:44733)
0924 11:41:23.545624 (+ 34432us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.545629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.551957 (+ 6328us) server_negotiation.cc:407] Connection header received
0924 11:41:23.552004 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.552009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.552096 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.552192 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.556376 (+ 4184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.557472 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.560707 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.561587 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.561738 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.566901 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.566906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.566911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.566946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.569909 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.570424 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.570427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.570429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.570570 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.570870 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.570873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.570874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.571091 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.571246 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.571462 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.571593 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34344,"thread_start_us":40107,"threads_started":1}
I20260924 11:41:23.575466 6534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:23.582119 6534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:23.606876 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.598575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39837 (local address 127.0.0.1:32888)
0924 11:41:23.599190 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.599210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.599327 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.601167 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.601171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.601193 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:23.601479 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.601487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.602683 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.602691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.604335 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.604351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.606114 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.606119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.606289 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.606324 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.606389 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.606507 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":313,"threads_started":1}
I20260924 11:41:23.607985 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.598740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.0.97.1:39837)
0924 11:41:23.600713 (+ 1973us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.600722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.600745 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:41:23.600829 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.600834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.601002 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.601182 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:23.601621 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.602558 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.604508 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.605906 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.607218 (+ 1312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.607422 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.607489 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.607675 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1812,"thread_start_us":1707,"threads_started":1}
I20260924 11:41:23.611346 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.598849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37021 (local address 127.0.0.1:38142)
0924 11:41:23.604604 (+ 5755us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.604625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.604731 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.605447 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.605452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.605473 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:23.605887 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.605899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.607644 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.607649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.609463 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.609476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.610578 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.610593 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.610746 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.610774 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.610907 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.611015 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5637,"thread_start_us":5504,"threads_started":1}
I20260924 11:41:23.611377 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.599349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.0.97.2:37021)
0924 11:41:23.599786 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.599794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.604716 (+ 4922us) server_negotiation.cc:407] Connection header received
0924 11:41:23.604861 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.604869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.605076 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.605270 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:23.606093 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.607510 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.609616 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.610402 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.610535 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.610712 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.610944 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.611100 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":219,"threads_started":1}
I20260924 11:41:23.611727 7436 tablet_service.cc:1467] Processing CreateTablet for tablet ead4d5c2ef9a4988a43a5d1946e65b9f (DEFAULT_TABLE table=test-table [id=50bf0ca339a44638b5c61c4f79141d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:23.613111 7436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4d5c2ef9a4988a43a5d1946e65b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:23.614495 7584 tablet_service.cc:1467] Processing CreateTablet for tablet ead4d5c2ef9a4988a43a5d1946e65b9f (DEFAULT_TABLE table=test-table [id=50bf0ca339a44638b5c61c4f79141d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:23.615353 7584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4d5c2ef9a4988a43a5d1946e65b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:23.617233 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.604941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.0.97.3:41155)
0924 11:41:23.605536 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.605545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.606666 (+ 1121us) server_negotiation.cc:407] Connection header received
0924 11:41:23.606903 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.606910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.607131 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.607294 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:23.609657 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.611063 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.614209 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.615521 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.615624 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.615797 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.616337 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.616489 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":309,"threads_started":1}
I20260924 11:41:23.617885 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.605519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41155 (local address 127.0.0.1:33264)
0924 11:41:23.606560 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.606577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.606705 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.607287 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.607291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.607315 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:23.607638 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.607647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.612254 (+ 4607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.612258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.614048 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.614061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.615958 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.615963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.616154 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.616183 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.617456 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.617586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":329,"threads_started":1}
I20260924 11:41:23.620996 7726 tablet_service.cc:1467] Processing CreateTablet for tablet ead4d5c2ef9a4988a43a5d1946e65b9f (DEFAULT_TABLE table=test-table [id=50bf0ca339a44638b5c61c4f79141d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:23.622143 7726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4d5c2ef9a4988a43a5d1946e65b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:23.627007 7825 tablet_bootstrap.cc:492] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Bootstrap starting.
I20260924 11:41:23.628789 7826 tablet_bootstrap.cc:492] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Bootstrap starting.
I20260924 11:41:23.630550 7825 tablet_bootstrap.cc:654] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:23.631824 7825 log.cc:824] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:23.632018 7826 tablet_bootstrap.cc:654] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:23.633555 7826 log.cc:824] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:23.635515 7825 tablet_bootstrap.cc:492] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: No bootstrap required, opened a new log
I20260924 11:41:23.635828 7825 ts_tablet_manager.cc:1397] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 11:41:23.635998 7829 tablet_bootstrap.cc:492] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Bootstrap starting.
I20260924 11:41:23.638792 7826 tablet_bootstrap.cc:492] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: No bootstrap required, opened a new log
I20260924 11:41:23.639019 7826 ts_tablet_manager.cc:1397] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260924 11:41:23.639139 7829 tablet_bootstrap.cc:654] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:23.640733 7829 log.cc:824] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:23.642908 7829 tablet_bootstrap.cc:492] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: No bootstrap required, opened a new log
I20260924 11:41:23.643131 7829 ts_tablet_manager.cc:1397] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260924 11:41:23.645730 7825 raft_consensus.cc:348] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.646143 7825 raft_consensus.cc:374] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:23.646229 7825 raft_consensus.cc:729] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99913db5b7e9468daf0f29cbb0971b4a, State: Initialized, Role: FOLLOWER
I20260924 11:41:23.646809 7825 consensus_queue.cc:260] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.648216 7650 heartbeater.cc:503] Master 127.0.97.62:44733 was elected leader, sending a full tablet report...
I20260924 11:41:23.648798 7825 ts_tablet_manager.cc:1428] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 11:41:23.651058 7826 raft_consensus.cc:348] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.651427 7826 raft_consensus.cc:374] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:23.651521 7826 raft_consensus.cc:729] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834abac14e68482495726b4d94f6f5d5, State: Initialized, Role: FOLLOWER
I20260924 11:41:23.651588 7829 raft_consensus.cc:348] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.651948 7829 raft_consensus.cc:374] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:23.652026 7829 raft_consensus.cc:729] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d468aa0894d49cf8c3ebb1997aaa21f, State: Initialized, Role: FOLLOWER
I20260924 11:41:23.652040 7826 consensus_queue.cc:260] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.652633 7829 consensus_queue.cc:260] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.655448 7826 ts_tablet_manager.cc:1428] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 11:41:23.659638 7829 ts_tablet_manager.cc:1428] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260924 11:41:23.659925 7792 heartbeater.cc:503] Master 127.0.97.62:44733 was elected leader, sending a full tablet report...
I20260924 11:41:23.678804 7832 raft_consensus.cc:482] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:23.678985 7832 raft_consensus.cc:504] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.680850 7832 leader_election.cc:288] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021), 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155)
I20260924 11:41:23.686627 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.681104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41721 (local address 127.0.97.2:37021)
0924 11:41:23.681309 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.681316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.681675 (+ 359us) server_negotiation.cc:407] Connection header received
0924 11:41:23.681903 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.681908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.681971 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.682026 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:23.682718 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.683356 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.685039 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.685744 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.685876 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.685968 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.686247 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.686367 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:41:23.686653 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.681701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41697 (local address 127.0.97.3:41155)
0924 11:41:23.681879 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.681884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.682130 (+ 246us) server_negotiation.cc:407] Connection header received
0924 11:41:23.682279 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.682282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.682328 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.682392 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:23.682929 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.683568 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.685144 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.685819 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.685950 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.686050 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.686333 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.686443 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:41:23.686699 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.681006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37021 (local address 127.0.97.1:41721)
0924 11:41:23.681556 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.681600 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.681741 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.682195 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.682201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.682241 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:23.682560 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.682583 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.683500 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.683505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.684901 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.684912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.685942 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.685948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.686094 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.686120 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.686207 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.686347 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":260,"threads_started":1}
I20260924 11:41:23.686949 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.681550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41155 (local address 127.0.97.1:41697)
0924 11:41:23.682039 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:23.682053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:23.682163 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:23.682426 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:23.682431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:23.682455 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:23.682784 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.682793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.683695 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.683700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.685010 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:23.685021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.686030 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.686036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:23.686167 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.686216 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 11:41:23.686300 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 11:41:23.686398 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":242,"threads_started":1}
I20260924 11:41:23.687525 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4d5c2ef9a4988a43a5d1946e65b9f" candidate_uuid: "834abac14e68482495726b4d94f6f5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99913db5b7e9468daf0f29cbb0971b4a" is_pre_election: true
I20260924 11:41:23.688035 7604 raft_consensus.cc:2454] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 834abac14e68482495726b4d94f6f5d5 in term 0.
I20260924 11:41:23.688292 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4d5c2ef9a4988a43a5d1946e65b9f" candidate_uuid: "834abac14e68482495726b4d94f6f5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" is_pre_election: true
I20260924 11:41:23.688654 7387 leader_election.cc:302] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 834abac14e68482495726b4d94f6f5d5, 99913db5b7e9468daf0f29cbb0971b4a; no voters:
I20260924 11:41:23.688709 7746 raft_consensus.cc:2454] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 834abac14e68482495726b4d94f6f5d5 in term 0.
I20260924 11:41:23.688974 7832 raft_consensus.cc:2781] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:23.689110 7832 raft_consensus.cc:482] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:23.689211 7832 raft_consensus.cc:3037] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:23.691269 7832 raft_consensus.cc:504] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.691748 7832 leader_election.cc:288] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [CANDIDATE]: Term 1 election: Requested vote from peers 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021), 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155)
I20260924 11:41:23.692273 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4d5c2ef9a4988a43a5d1946e65b9f" candidate_uuid: "834abac14e68482495726b4d94f6f5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f"
I20260924 11:41:23.692451 7746 raft_consensus.cc:3037] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:23.692451 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4d5c2ef9a4988a43a5d1946e65b9f" candidate_uuid: "834abac14e68482495726b4d94f6f5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99913db5b7e9468daf0f29cbb0971b4a"
I20260924 11:41:23.692627 7604 raft_consensus.cc:3037] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:23.695487 7604 raft_consensus.cc:2454] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 834abac14e68482495726b4d94f6f5d5 in term 1.
I20260924 11:41:23.695487 7746 raft_consensus.cc:2454] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 834abac14e68482495726b4d94f6f5d5 in term 1.
I20260924 11:41:23.695789 7387 leader_election.cc:302] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d468aa0894d49cf8c3ebb1997aaa21f, 834abac14e68482495726b4d94f6f5d5; no voters:
I20260924 11:41:23.696018 7832 raft_consensus.cc:2781] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:23.696426 7832 raft_consensus.cc:686] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 1 LEADER]: Becoming Leader. State: Replica: 834abac14e68482495726b4d94f6f5d5, State: Running, Role: LEADER
I20260924 11:41:23.696914 7832 consensus_queue.cc:237] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } }
I20260924 11:41:23.701754 6534 catalog_manager.cc:5495] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 834abac14e68482495726b4d94f6f5d5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "834abac14e68482495726b4d94f6f5d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } health_report { overall_health: UNKNOWN } } }
W20260924 11:41:23.710522 7652 tablet.cc:2369] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:23.726898 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.722011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.0.97.1:39837)
0924 11:41:23.722183 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.722190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.722212 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:41:23.722430 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.722435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.722502 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.722649 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:23.722971 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.723611 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.725034 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.725413 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.725540 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.725952 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:23.726359 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:23.726507 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.726559 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.726625 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:41:23.733731 7746 raft_consensus.cc:1261] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Refusing update from remote peer 834abac14e68482495726b4d94f6f5d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:23.733794 7604 raft_consensus.cc:1261] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Refusing update from remote peer 834abac14e68482495726b4d94f6f5d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:23.734562 7836 consensus_queue.cc:1035] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:23.734894 7832 consensus_queue.cc:1035] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:23.746444 7839 mvcc.cc:205] Tried to move back new op lower bound from 7332864342965547008 to 7332864342825996288. Current Snapshot: MvccSnapshot[applied={T|T < 7332864342965547008}]
I20260924 11:41:23.746974 7840 mvcc.cc:205] Tried to move back new op lower bound from 7332864342965547008 to 7332864342825996288. Current Snapshot: MvccSnapshot[applied={T|T < 7332864342965547008}]
I20260924 11:41:23.759087 7841 mvcc.cc:205] Tried to move back new op lower bound from 7332864342965547008 to 7332864342825996288. Current Snapshot: MvccSnapshot[applied={T|T < 7332864342965547008}]
I20260924 11:41:23.765579 6534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:23.765802 6534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:23.771713 6534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57cdef02ecfa4c998ae8a9e3f5c41170: Sending DeleteTablet for 3 replicas of tablet ead4d5c2ef9a4988a43a5d1946e65b9f
I20260924 11:41:23.772617 7436 tablet_service.cc:1514] Processing DeleteTablet for tablet ead4d5c2ef9a4988a43a5d1946e65b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32888
I20260924 11:41:23.773387 7850 tablet_replica.cc:330] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: stopping tablet replica
I20260924 11:41:23.773959 7850 raft_consensus.cc:2229] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:23.774101 7584 tablet_service.cc:1514] Processing DeleteTablet for tablet ead4d5c2ef9a4988a43a5d1946e65b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38142
I20260924 11:41:23.774760 7726 tablet_service.cc:1514] Processing DeleteTablet for tablet ead4d5c2ef9a4988a43a5d1946e65b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33264
I20260924 11:41:23.774895 7851 tablet_replica.cc:330] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: stopping tablet replica
I20260924 11:41:23.775206 7851 raft_consensus.cc:2229] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:23.775663 7851 raft_consensus.cc:2258] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:23.777350 7851 ts_tablet_manager.cc:1905] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:23.782491 7850 raft_consensus.cc:2258] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:23.784016 7851 ts_tablet_manager.cc:1918] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:23.784168 7851 log.cc:1196] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/ead4d5c2ef9a4988a43a5d1946e65b9f
I20260924 11:41:23.784646 7851 ts_tablet_manager.cc:1939] T ead4d5c2ef9a4988a43a5d1946e65b9f P 99913db5b7e9468daf0f29cbb0971b4a: Deleting consensus metadata
I20260924 11:41:23.784710 7853 tablet_replica.cc:330] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: stopping tablet replica
I20260924 11:41:23.785116 7853 raft_consensus.cc:2229] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:23.785598 7853 raft_consensus.cc:2258] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:23.786273 6520 catalog_manager.cc:4841] TS 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021): tablet ead4d5c2ef9a4988a43a5d1946e65b9f (table test-table [id=50bf0ca339a44638b5c61c4f79141d91]) successfully deleted
I20260924 11:41:23.787580 7850 ts_tablet_manager.cc:1905] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:23.787748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6503
I20260924 11:41:23.788719 6565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:23.790566 7853 ts_tablet_manager.cc:1905] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:23.797298 7853 ts_tablet_manager.cc:1918] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:23.797305 7850 ts_tablet_manager.cc:1918] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:23.797442 7853 log.cc:1196] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/ead4d5c2ef9a4988a43a5d1946e65b9f
I20260924 11:41:23.797554 7850 log.cc:1196] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/ead4d5c2ef9a4988a43a5d1946e65b9f
I20260924 11:41:23.797972 7853 ts_tablet_manager.cc:1939] T ead4d5c2ef9a4988a43a5d1946e65b9f P 5d468aa0894d49cf8c3ebb1997aaa21f: Deleting consensus metadata
I20260924 11:41:23.798164 7850 ts_tablet_manager.cc:1939] T ead4d5c2ef9a4988a43a5d1946e65b9f P 834abac14e68482495726b4d94f6f5d5: Deleting consensus metadata
I20260924 11:41:23.809188 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.776134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.0.97.61:42233)
0924 11:41:23.776341 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.776349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.786791 (+ 10442us) server_negotiation.cc:407] Connection header received
0924 11:41:23.786968 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.786973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.787048 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.787147 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.799432 (+ 12285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.800093 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.801331 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.801749 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.801867 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.804178 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.804184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.804189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.804220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.806670 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.807300 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.807307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.807310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.807428 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.807777 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.807784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.807787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.808027 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.808287 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.808585 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.808831 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:41:23.820851 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.786356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.0.97.60:39555)
0924 11:41:23.786565 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.786571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.809376 (+ 22805us) server_negotiation.cc:407] Connection header received
0924 11:41:23.809812 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.809817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.809899 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.809977 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.811247 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.811899 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.813301 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.813666 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.813786 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:23.816001 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:23.816005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:23.816008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:23.816036 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:23.818530 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.819093 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.819098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.819122 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.819234 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:23.819574 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:23.819579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:23.819582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:23.819820 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:23.820066 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:23.820340 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 11:41:23.820539 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:41:23.954200 6520 catalog_manager.cc:4841] TS 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155): tablet ead4d5c2ef9a4988a43a5d1946e65b9f (table test-table [id=50bf0ca339a44638b5c61c4f79141d91]) successfully deleted
I20260924 11:41:23.955369 6522 catalog_manager.cc:4841] TS 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837): tablet ead4d5c2ef9a4988a43a5d1946e65b9f (table test-table [id=50bf0ca339a44638b5c61c4f79141d91]) successfully deleted
I20260924 11:41:23.956036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6503
I20260924 11:41:24.007936 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.981625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.0.97.61:42233)
0924 11:41:23.981795 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.981801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.992476 (+ 10675us) server_negotiation.cc:407] Connection header received
0924 11:41:23.992718 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.992724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.992804 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.992871 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.995065 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.996145 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.998223 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.998849 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:23.998961 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:24.001809 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:24.001814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:24.001818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:24.001851 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:24.005114 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:24.005919 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:24.005924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:24.005927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:24.006057 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:24.006519 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:24.006524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:24.006528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:24.006835 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:24.007086 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:24.007437 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 11:41:24.007657 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:41:24.011053 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:23.981828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.0.97.60:39555)
0924 11:41:23.982055 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:23.982065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:23.994334 (+ 12269us) server_negotiation.cc:407] Connection header received
0924 11:41:23.994877 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:23.994886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:23.995029 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:23.995119 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:23.998691 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:23.999695 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:24.001613 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:24.002059 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:24.002184 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:24.004715 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:24.004726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:24.004729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:24.004758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:24.008558 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:24.009257 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:24.009261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:24.009264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:24.009389 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:24.009779 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:24.009783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:24.009785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:24.010067 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:24.010227 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:24.010561 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 11:41:24.010700 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:41:24.754931 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:24.754070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:33615)
0924 11:41:24.754509 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:24.754646 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":459,"threads_started":1}
W20260924 11:41:24.755416 7792 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:24.761396 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:24.760740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:47721)
0924 11:41:24.761123 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:24.761252 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":215,"threads_started":1}
W20260924 11:41:24.761687 7502 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:24.765857 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:24.765292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:41845)
0924 11:41:24.765636 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:24.765722 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":155,"threads_started":1}
W20260924 11:41:24.766158 7650 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:25.271723 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57cdef02ecfa4c998ae8a9e3f5c41170)
I20260924 11:41:25.271908 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } }
I20260924 11:41:25.272631 7865 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733), 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61:42233)
I20260924 11:41:25.273648 7869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57cdef02ecfa4c998ae8a9e3f5c41170)
I20260924 11:41:25.273881 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.272768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44070)
0924 11:41:25.273406 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.273630 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"threads_started":1}
I20260924 11:41:25.274025 7869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } }
W20260924 11:41:25.274518 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:25.275750 7869 leader_election.cc:288] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733), 4ff056c2d472434aae750f62f7b310c3 (127.0.97.60:39555)
I20260924 11:41:25.276510 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.275992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44084)
0924 11:41:25.276177 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.276314 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 11:41:25.276702 7308 leader_election.cc:334] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:25.277127 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.275810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44078)
0924 11:41:25.276481 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.276717 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":502,"threads_started":1}
W20260924 11:41:25.277480 7120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:25.279050 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.278747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44096)
0924 11:41:25.278869 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.278965 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 11:41:25.279244 7120 leader_election.cc:334] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:25.282294 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.272926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.0.97.61:42233)
0924 11:41:25.275073 (+ 2147us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.275079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.275098 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:41:25.275159 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.275163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.275248 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.275351 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.276878 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.278345 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.280524 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.281379 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.281513 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.281609 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.281877 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.281993 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":2035,"thread_start_us":1862,"threads_started":1}
I20260924 11:41:25.282323 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.272795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42233 (local address 127.0.0.1:55526)
0924 11:41:25.274369 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.274391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.274553 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.275397 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.275401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.275448 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.275700 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.275714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.278496 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.278502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.280397 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.280407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.281559 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.281565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.281733 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.281755 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.281857 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.281982 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":352,"threads_started":1}
I20260924 11:41:25.282887 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff056c2d472434aae750f62f7b310c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8898367107cd49fb8bd6dccff975f1d3" is_pre_election: true
I20260924 11:41:25.283114 7143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff056c2d472434aae750f62f7b310c3 in term 1.
I20260924 11:41:25.283272 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.276180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.97.60:39555)
0924 11:41:25.277402 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.277409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.277430 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:25.277504 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.277509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.277568 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.277718 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.278591 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.279833 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.281460 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.282240 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.282386 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.282484 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.282851 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.282963 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":1110,"thread_start_us":1005,"threads_started":1}
I20260924 11:41:25.283376 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.275909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39555 (local address 127.0.0.1:38442)
0924 11:41:25.276529 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.276544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.276977 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.277733 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.277737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.277762 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.278310 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.278324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.279951 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.279955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.281316 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.281326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.282577 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.282585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.282708 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.282729 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.282831 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.282947 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":2,"thread_start_us":763,"threads_started":1}
I20260924 11:41:25.283634 7306 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ff056c2d472434aae750f62f7b310c3, 8898367107cd49fb8bd6dccff975f1d3; no voters: 57cdef02ecfa4c998ae8a9e3f5c41170
I20260924 11:41:25.283962 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 11:41:25.283958 7331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8898367107cd49fb8bd6dccff975f1d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ff056c2d472434aae750f62f7b310c3" is_pre_election: true
I20260924 11:41:25.284096 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 57cdef02ecfa4c998ae8a9e3f5c41170)
I20260924 11:41:25.284250 7865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:41:25.286270 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } }
I20260924 11:41:25.286693 7331 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8898367107cd49fb8bd6dccff975f1d3 in current term 2: Already voted for candidate 4ff056c2d472434aae750f62f7b310c3 in this term.
I20260924 11:41:25.286863 7865 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 2 election: Requested vote from peers 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733), 8898367107cd49fb8bd6dccff975f1d3 (127.0.97.61:42233)
I20260924 11:41:25.287390 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.287006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44104)
0924 11:41:25.287118 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.287176 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 11:41:25.287587 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff056c2d472434aae750f62f7b310c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8898367107cd49fb8bd6dccff975f1d3"
I20260924 11:41:25.287695 7119 leader_election.cc:302] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8898367107cd49fb8bd6dccff975f1d3; no voters: 4ff056c2d472434aae750f62f7b310c3, 57cdef02ecfa4c998ae8a9e3f5c41170
I20260924 11:41:25.287804 7143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:41:25.288300 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.287898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44112)
0924 11:41:25.288075 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.288189 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 11:41:25.288475 7308 leader_election.cc:334] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
I20260924 11:41:25.289991 7143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff056c2d472434aae750f62f7b310c3 in term 2.
I20260924 11:41:25.290190 7869 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:41:25.290354 7306 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ff056c2d472434aae750f62f7b310c3, 8898367107cd49fb8bd6dccff975f1d3; no voters: 57cdef02ecfa4c998ae8a9e3f5c41170
I20260924 11:41:25.290648 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:41:25.291039 7865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [term 2 LEADER]: Becoming Leader. State: Replica: 4ff056c2d472434aae750f62f7b310c3, State: Running, Role: LEADER
I20260924 11:41:25.291388 7865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } }
I20260924 11:41:25.293169 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff056c2d472434aae750f62f7b310c3. Latest consensus state: current_term: 2 leader_uuid: "4ff056c2d472434aae750f62f7b310c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:25.293511 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:25.294062 7877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:25.297407 7877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:25.298188 7877 catalog_manager.cc:1259] Loaded cluster ID: e47f00f057ad4e95bb918af8deb82a2b
I20260924 11:41:25.298278 7877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:25.299328 7877 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:25.300052 7877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Loaded TSK: 0
I20260924 11:41:25.300833 7877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:41:25.319406 7321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:25.325970 7321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:25.329684 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [term 2 FOLLOWER]: Refusing update from remote peer 4ff056c2d472434aae750f62f7b310c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 11:41:25.329717 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.329278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44132)
0924 11:41:25.329451 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.329527 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 11:41:25.330449 7875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 11:41:25.330804 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.330466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44136)
0924 11:41:25.330600 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.330677 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:41:25.330991 7308 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 -> Peer 57cdef02ecfa4c998ae8a9e3f5c41170 (127.0.97.62:44733): Couldn't send request to peer 57cdef02ecfa4c998ae8a9e3f5c41170. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 11:41:25.335238 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff056c2d472434aae750f62f7b310c3. Latest consensus state: current_term: 2 leader_uuid: "4ff056c2d472434aae750f62f7b310c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:25.335471 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:41:25.336796 7865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4ff056c2d472434aae750f62f7b310c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:25.337225 7865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:25.342481 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4ff056c2d472434aae750f62f7b310c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57cdef02ecfa4c998ae8a9e3f5c41170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44733 } } peers { permanent_uuid: "8898367107cd49fb8bd6dccff975f1d3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "4ff056c2d472434aae750f62f7b310c3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39555 } attrs { promote: false } } }
I20260924 11:41:25.342739 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8898367107cd49fb8bd6dccff975f1d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:41:25.358189 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.346333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41155 (local address 127.0.0.1:33268)
0924 11:41:25.346988 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.347007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.347115 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.352704 (+ 5589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.352708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.352731 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.352907 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.352921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.354268 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.354273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.356255 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.356268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.357422 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.357428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.357622 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.357645 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.357761 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.357886 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515}
I20260924 11:41:25.358189 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.346446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.0.97.3:41155)
0924 11:41:25.352438 (+ 5992us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.352443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.352457 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:41:25.352510 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.352513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.352574 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.352690 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.353069 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.354136 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.356400 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.357271 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.357400 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.357511 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.357786 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.357909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5889,"thread_start_us":5811,"threads_started":1}
I20260924 11:41:25.359229 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.346106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39837 (local address 127.0.0.1:32900)
0924 11:41:25.346523 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.346547 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.346686 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.353401 (+ 6715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.353408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.353442 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.353785 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.353801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.355107 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.355113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.357080 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.357092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.358686 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.358692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.358845 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.358865 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.358934 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.359018 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260924 11:41:25.359076 7726 tablet_service.cc:1467] Processing CreateTablet for tablet b7e12d04b4704d8d8c92e23a2fba5b93 (DEFAULT_TABLE table=test-table [id=4e81d54a14b74c1286ea7ba1523f3915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:25.359441 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.346211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.0.97.1:39837)
0924 11:41:25.353109 (+ 6898us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.353115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.353132 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:41:25.353195 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.353199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.353260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.353353 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.353989 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.354958 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.357242 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.358420 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.358622 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.358717 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.358983 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.359159 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6786,"thread_start_us":6693,"threads_started":1}
I20260924 11:41:25.359575 7726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e12d04b4704d8d8c92e23a2fba5b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:25.360216 7436 tablet_service.cc:1467] Processing CreateTablet for tablet b7e12d04b4704d8d8c92e23a2fba5b93 (DEFAULT_TABLE table=test-table [id=4e81d54a14b74c1286ea7ba1523f3915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:25.360642 7436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e12d04b4704d8d8c92e23a2fba5b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:25.361785 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.346930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37021 (local address 127.0.0.1:38152)
0924 11:41:25.354140 (+ 7210us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.354157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.354267 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.354592 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.354596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.354616 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.354954 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.354962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.356705 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.356709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.359510 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.359527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.361006 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.361012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.361185 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.361215 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.361299 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.361417 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7094,"thread_start_us":6975,"threads_started":1}
I20260924 11:41:25.362350 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.347363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.0.97.2:37021)
0924 11:41:25.353778 (+ 6415us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.353783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.354368 (+ 585us) server_negotiation.cc:407] Connection header received
0924 11:41:25.354425 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.354428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.354479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.354576 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.355105 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.356580 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.359677 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.360803 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.360934 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.361043 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.361963 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.362100 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6305,"thread_start_us":6190,"threads_started":1}
I20260924 11:41:25.364675 7887 tablet_bootstrap.cc:492] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Bootstrap starting.
I20260924 11:41:25.364866 7584 tablet_service.cc:1467] Processing CreateTablet for tablet b7e12d04b4704d8d8c92e23a2fba5b93 (DEFAULT_TABLE table=test-table [id=4e81d54a14b74c1286ea7ba1523f3915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:25.365275 7584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e12d04b4704d8d8c92e23a2fba5b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:25.365821 7888 tablet_bootstrap.cc:492] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Bootstrap starting.
I20260924 11:41:25.366297 7887 tablet_bootstrap.cc:654] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:25.367417 7888 tablet_bootstrap.cc:654] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:25.369069 7887 tablet_bootstrap.cc:492] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: No bootstrap required, opened a new log
I20260924 11:41:25.369230 7887 ts_tablet_manager.cc:1397] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 11:41:25.370030 7887 raft_consensus.cc:348] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.370258 7887 raft_consensus.cc:374] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:25.370364 7887 raft_consensus.cc:729] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d468aa0894d49cf8c3ebb1997aaa21f, State: Initialized, Role: FOLLOWER
I20260924 11:41:25.370613 7887 consensus_queue.cc:260] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.371302 7891 tablet_bootstrap.cc:492] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Bootstrap starting.
I20260924 11:41:25.371469 7887 ts_tablet_manager.cc:1428] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:25.373037 7888 tablet_bootstrap.cc:492] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: No bootstrap required, opened a new log
I20260924 11:41:25.373209 7888 ts_tablet_manager.cc:1397] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260924 11:41:25.374120 7888 raft_consensus.cc:348] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.374387 7888 raft_consensus.cc:374] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:25.374500 7888 raft_consensus.cc:729] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834abac14e68482495726b4d94f6f5d5, State: Initialized, Role: FOLLOWER
I20260924 11:41:25.374716 7888 consensus_queue.cc:260] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.374969 7791 heartbeater.cc:503] Master 127.0.97.60:39555 was elected leader, sending a full tablet report...
I20260924 11:41:25.378046 7891 tablet_bootstrap.cc:654] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:25.380513 7891 tablet_bootstrap.cc:492] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: No bootstrap required, opened a new log
I20260924 11:41:25.380625 7888 ts_tablet_manager.cc:1428] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260924 11:41:25.380677 7891 ts_tablet_manager.cc:1397] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260924 11:41:25.380950 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.371905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:35575)
0924 11:41:25.380555 (+ 8650us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.380718 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8545,"thread_start_us":8429,"threads_started":1}
I20260924 11:41:25.381922 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.381624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:52463)
0924 11:41:25.381769 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.381834 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:41:25.382407 7891 raft_consensus.cc:348] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.382690 7891 raft_consensus.cc:374] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:25.383059 7891 raft_consensus.cc:729] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99913db5b7e9468daf0f29cbb0971b4a, State: Initialized, Role: FOLLOWER
I20260924 11:41:25.383898 7891 consensus_queue.cc:260] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.384414 7501 heartbeater.cc:503] Master 127.0.97.60:39555 was elected leader, sending a full tablet report...
I20260924 11:41:25.385304 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.381136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:47857)
0924 11:41:25.385062 (+ 3926us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.385151 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3805,"threads_started":1}
I20260924 11:41:25.386509 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.386258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:49861)
0924 11:41:25.386363 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.386429 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 11:41:25.391964 7891 ts_tablet_manager.cc:1428] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260924 11:41:25.393141 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.392282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:35649)
0924 11:41:25.392795 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.392912 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":240,"threads_started":1}
I20260924 11:41:25.393484 7649 heartbeater.cc:503] Master 127.0.97.60:39555 was elected leader, sending a full tablet report...
I20260924 11:41:25.394435 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.394085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:49255)
0924 11:41:25.394234 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.394311 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 11:41:25.508292 7794 tablet.cc:2369] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:25.620733 7892 raft_consensus.cc:482] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:25.620901 7892 raft_consensus.cc:504] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.622263 7892 leader_election.cc:288] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837), 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021)
W20260924 11:41:25.626751 7511 tablet.cc:2369] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:25.628319 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.622625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37899 (local address 127.0.97.1:39837)
0924 11:41:25.622876 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.622882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.623176 (+ 294us) server_negotiation.cc:407] Connection header received
0924 11:41:25.623326 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.623330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.623382 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.623483 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.624092 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.624763 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.626405 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.627387 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.627521 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.627628 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.627927 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.628040 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 11:41:25.628502 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.622536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39837 (local address 127.0.97.3:37899)
0924 11:41:25.623079 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.623095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.623211 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.623511 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.623517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.623556 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.623937 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.623946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.624904 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.624907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.626270 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.626281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.627548 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.627554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.627721 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.627802 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.627904 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.628024 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":77,"spinlock_wait_cycles":7680,"thread_start_us":288,"threads_started":1}
I20260924 11:41:25.628504 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.623004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35667 (local address 127.0.97.2:37021)
0924 11:41:25.623222 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.623228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.623246 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:41:25.623315 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.623320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.623380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.623465 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:25.623993 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.624742 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.626417 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.627636 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.627771 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.627865 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.628169 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.628282 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:41:25.628561 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.622479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37021 (local address 127.0.97.3:35667)
0924 11:41:25.622649 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.622669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:25.622824 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:25.623489 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:25.623494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:25.623516 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:25.623812 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.623820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.624870 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.624874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.626267 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:25.626278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.627814 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.627820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:25.627994 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.628015 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:41:25.628127 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 11:41:25.628283 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":6,"spinlock_wait_cycles":1920}
I20260924 11:41:25.629323 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12d04b4704d8d8c92e23a2fba5b93" candidate_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99913db5b7e9468daf0f29cbb0971b4a" is_pre_election: true
I20260924 11:41:25.629391 7456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12d04b4704d8d8c92e23a2fba5b93" candidate_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834abac14e68482495726b4d94f6f5d5" is_pre_election: true
I20260924 11:41:25.629572 7604 raft_consensus.cc:2454] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d468aa0894d49cf8c3ebb1997aaa21f in term 0.
I20260924 11:41:25.629778 7456 raft_consensus.cc:2454] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d468aa0894d49cf8c3ebb1997aaa21f in term 0.
I20260924 11:41:25.630097 7680 leader_election.cc:302] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d468aa0894d49cf8c3ebb1997aaa21f, 99913db5b7e9468daf0f29cbb0971b4a; no voters:
I20260924 11:41:25.630496 7892 raft_consensus.cc:2781] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:25.630604 7892 raft_consensus.cc:482] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:25.630726 7892 raft_consensus.cc:3037] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:25.632745 7892 raft_consensus.cc:504] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.633291 7892 leader_election.cc:288] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [CANDIDATE]: Term 1 election: Requested vote from peers 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837), 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021)
I20260924 11:41:25.633612 7456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12d04b4704d8d8c92e23a2fba5b93" candidate_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834abac14e68482495726b4d94f6f5d5"
I20260924 11:41:25.633765 7456 raft_consensus.cc:3037] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:25.633883 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12d04b4704d8d8c92e23a2fba5b93" candidate_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99913db5b7e9468daf0f29cbb0971b4a"
I20260924 11:41:25.634094 7604 raft_consensus.cc:3037] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:25.636799 7456 raft_consensus.cc:2454] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d468aa0894d49cf8c3ebb1997aaa21f in term 1.
I20260924 11:41:25.636889 7604 raft_consensus.cc:2454] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d468aa0894d49cf8c3ebb1997aaa21f in term 1.
I20260924 11:41:25.637235 7682 leader_election.cc:302] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d468aa0894d49cf8c3ebb1997aaa21f, 834abac14e68482495726b4d94f6f5d5, 99913db5b7e9468daf0f29cbb0971b4a; no voters:
I20260924 11:41:25.637528 7892 raft_consensus.cc:2781] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:25.637933 7892 raft_consensus.cc:686] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 LEADER]: Becoming Leader. State: Replica: 5d468aa0894d49cf8c3ebb1997aaa21f, State: Running, Role: LEADER
I20260924 11:41:25.638871 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.638466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:36153)
0924 11:41:25.638607 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.638735 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:41:25.638710 7892 consensus_queue.cc:237] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } }
I20260924 11:41:25.642236 7320 catalog_manager.cc:5495] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d468aa0894d49cf8c3ebb1997aaa21f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 } health_report { overall_health: UNKNOWN } } }
I20260924 11:41:25.707311 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.706973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44152)
0924 11:41:25.707118 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.707195 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:41:25.708324 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.707909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44154)
0924 11:41:25.708034 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.708149 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 11:41:25.715152 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.710475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.97.3:41155)
0924 11:41:25.710659 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.710664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.710681 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:41:25.710892 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.710896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.710962 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.711134 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:25.711453 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.712115 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.713299 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.713674 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.713791 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.714176 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:25.714632 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:25.714778 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.714847 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.714911 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:41:25.718707 7604 raft_consensus.cc:1261] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Refusing update from remote peer 5d468aa0894d49cf8c3ebb1997aaa21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:25.719043 7456 raft_consensus.cc:1261] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Refusing update from remote peer 5d468aa0894d49cf8c3ebb1997aaa21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:25.719309 7900 consensus_queue.cc:1035] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99913db5b7e9468daf0f29cbb0971b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:25.719683 7892 consensus_queue.cc:1035] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "834abac14e68482495726b4d94f6f5d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:25.721776 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.721251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:34545)
0924 11:41:25.721507 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.721612 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 11:41:25.723245 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.721277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:60005)
0924 11:41:25.722990 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.723070 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1606}
I20260924 11:41:25.728359 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.727961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.1:60881)
0924 11:41:25.728126 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.728238 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 11:41:25.728832 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.728276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.3:36129)
0924 11:41:25.728593 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.728686 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260924 11:41:25.733384 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.732981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.97.2:53657)
0924 11:41:25.733180 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:25.733248 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 11:41:25.740028 7321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:25.740301 7321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:25.745069 7321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ff056c2d472434aae750f62f7b310c3: Sending DeleteTablet for 3 replicas of tablet b7e12d04b4704d8d8c92e23a2fba5b93
I20260924 11:41:25.745826 7436 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e12d04b4704d8d8c92e23a2fba5b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:25 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:32900
I20260924 11:41:25.746647 7913 tablet_replica.cc:330] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: stopping tablet replica
I20260924 11:41:25.746614 7726 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e12d04b4704d8d8c92e23a2fba5b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:25 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33268
I20260924 11:41:25.746850 7913 raft_consensus.cc:2229] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:25.747236 7913 raft_consensus.cc:2258] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:25.747265 7914 tablet_replica.cc:330] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: stopping tablet replica
I20260924 11:41:25.747431 7914 raft_consensus.cc:2229] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:25.747956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 7298
I20260924 11:41:25.747956 7914 raft_consensus.cc:2258] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:25.748414 7913 ts_tablet_manager.cc:1905] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:25.749408 7914 ts_tablet_manager.cc:1905] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:25.750782 7584 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e12d04b4704d8d8c92e23a2fba5b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:25 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38152
I20260924 11:41:25.755661 7913 ts_tablet_manager.cc:1918] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:25.755877 7913 log.cc:1196] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/b7e12d04b4704d8d8c92e23a2fba5b93
I20260924 11:41:25.756505 7913 ts_tablet_manager.cc:1939] T b7e12d04b4704d8d8c92e23a2fba5b93 P 834abac14e68482495726b4d94f6f5d5: Deleting consensus metadata
I20260924 11:41:25.757756 7308 catalog_manager.cc:4841] TS 834abac14e68482495726b4d94f6f5d5 (127.0.97.1:39837): tablet b7e12d04b4704d8d8c92e23a2fba5b93 (table test-table [id=4e81d54a14b74c1286ea7ba1523f3915]) successfully deleted
I20260924 11:41:25.758697 7914 ts_tablet_manager.cc:1918] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:25.758826 7914 log.cc:1196] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/b7e12d04b4704d8d8c92e23a2fba5b93
I20260924 11:41:25.759282 7914 ts_tablet_manager.cc:1939] T b7e12d04b4704d8d8c92e23a2fba5b93 P 5d468aa0894d49cf8c3ebb1997aaa21f: Deleting consensus metadata
I20260924 11:41:25.760286 7306 catalog_manager.cc:4841] TS 5d468aa0894d49cf8c3ebb1997aaa21f (127.0.97.3:41155): tablet b7e12d04b4704d8d8c92e23a2fba5b93 (table test-table [id=4e81d54a14b74c1286ea7ba1523f3915]) successfully deleted
I20260924 11:41:25.761596 7916 tablet_replica.cc:330] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: stopping tablet replica
I20260924 11:41:25.761770 7916 raft_consensus.cc:2229] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:25.761998 7916 raft_consensus.cc:2258] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:25.762820 7916 ts_tablet_manager.cc:1905] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:25.768929 7916 ts_tablet_manager.cc:1918] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:25.769094 7916 log.cc:1196] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/b7e12d04b4704d8d8c92e23a2fba5b93
I20260924 11:41:25.769562 7916 ts_tablet_manager.cc:1939] T b7e12d04b4704d8d8c92e23a2fba5b93 P 99913db5b7e9468daf0f29cbb0971b4a: Deleting consensus metadata
I20260924 11:41:25.770509 7306 catalog_manager.cc:4841] TS 99913db5b7e9468daf0f29cbb0971b4a (127.0.97.2:37021): tablet b7e12d04b4704d8d8c92e23a2fba5b93 (table test-table [id=4e81d54a14b74c1286ea7ba1523f3915]) successfully deleted
I20260924 11:41:25.777719 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.748604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.97.1:39837)
0924 11:41:25.748770 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.748777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.764521 (+ 15744us) server_negotiation.cc:407] Connection header received
0924 11:41:25.764709 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.764715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.764780 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.764871 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:25.766122 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.766840 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.767984 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.768437 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.768554 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.771403 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:25.771417 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:25.771433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:25.771469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:25.775192 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:25.775775 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:25.775784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:25.775788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:25.775929 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:25.776306 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:25.776310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:25.776314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:25.776725 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:25.776996 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.777185 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.777381 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:41:25.778277 7496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:25.892848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 7298
I20260924 11:41:25.921790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 7513
I20260924 11:41:25.938812 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:25.922539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.97.2:37021)
0924 11:41:25.922716 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:25.922720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:25.922783 (+ 63us) server_negotiation.cc:407] Connection header received
0924 11:41:25.922961 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:25.922965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:25.923042 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:25.923135 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:25.924821 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.925525 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.927338 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:25.927991 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:25.928164 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:25.931421 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:25.931438 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:25.931457 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:25.931494 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:25.935502 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:25.936255 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:25.936265 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:25.936270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:25.936415 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:25.936786 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:25.936791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:25.936794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:25.937220 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:25.937531 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:25.938117 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 11:41:25.938408 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:41:25.939574 7644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:26.027108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 7513
I20260924 11:41:26.045408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 7662
I20260924 11:41:26.060534 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:26.046206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.97.3:41155)
0924 11:41:26.046387 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:26.046393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:26.046484 (+ 91us) server_negotiation.cc:407] Connection header received
0924 11:41:26.046662 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:26.046665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:26.046736 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:26.046836 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:26.048676 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:26.049438 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:26.051134 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:26.051485 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:26.051565 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:26.053847 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:26.053859 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:26.053877 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:26.053911 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:26.057718 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:26.058328 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:26.058338 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:26.058341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:26.058541 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:26.058860 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:26.058865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:26.058869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:26.059289 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:26.059594 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:26.059859 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 11:41:26.060064 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:41:26.061298 7786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:26.105931 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:26.105531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44162)
0924 11:41:26.105702 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:26.105795 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:41:26.106817 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:26.106480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44733 (local address 127.0.0.1:44168)
0924 11:41:26.106641 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:26.106699 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 11:41:26.181375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 7662
I20260924 11:41:26.201843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 6996
I20260924 11:41:26.202837 7163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:26.313997 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 6996
I20260924 11:41:26.330250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 7097
I20260924 11:41:26.331182 7351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:26.442054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 7097
2026-09-24T11:41:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9025 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:26 dist-test-slave-qhrh krb5kdc[7924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:26 dist-test-slave-qhrh krb5kdc[7924](info): set up 2 sockets
Sep 24 11:41:26 dist-test-slave-qhrh krb5kdc[7924](info): commencing operation
krb5kdc: starting...
W20260924 11:41:27.504739 388 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:27 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 11:41:27.687429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40765
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:27.982195 7937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:27.982436 7937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:27.982517 7937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:27.989281 7937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:27.989359 7937 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:41:27.989424 7937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:27.989529 7937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:27.989604 7937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:27.989732 7937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:27.998594 7937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40765
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:27.999742 7937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:28.001027 7937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:28.011787 7942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:28.011873 7943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:28.011893 7945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:28.013082 7937 server_base.cc:1034] running on GCE node
I20260924 11:41:28.013742 7937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:28.014992 7937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:28.016261 7937 hybrid_clock.cc:648] HybridClock initialized: now 1790250088016205 us; error 46 us; skew 500 ppm
I20260924 11:41:28.016569 7937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:28 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250088, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:28.030943 7937 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 11:41:28.032356 7937 webserver.cc:460] Webserver started at http://127.0.97.62:43539/ using document root <none> and password file <none>
I20260924 11:41:28.032976 7937 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:28.033054 7937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:28.033301 7937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:28.035944 7937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "cda07ce6a2434a7695e6316874cc0f14"
format_stamp: "Formatted at 2026-09-24 11:41:28 on dist-test-slave-qhrh"
server_key: "7b138cace54d94aca644214618cc2592"
server_key_iv: "125a0f3e8188709e08004242fb24a72d"
server_key_version: "encryptionkey@0"
I20260924 11:41:28.036711 7937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "cda07ce6a2434a7695e6316874cc0f14"
format_stamp: "Formatted at 2026-09-24 11:41:28 on dist-test-slave-qhrh"
server_key: "7b138cace54d94aca644214618cc2592"
server_key_iv: "125a0f3e8188709e08004242fb24a72d"
server_key_version: "encryptionkey@0"
I20260924 11:41:28.041283 7937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:41:28.044476 7952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:28.045909 7937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:41:28.046067 7937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "cda07ce6a2434a7695e6316874cc0f14"
format_stamp: "Formatted at 2026-09-24 11:41:28 on dist-test-slave-qhrh"
server_key: "7b138cace54d94aca644214618cc2592"
server_key_iv: "125a0f3e8188709e08004242fb24a72d"
server_key_version: "encryptionkey@0"
I20260924 11:41:28.046326 7937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:28.064333 7937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:28.068380 7937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:28.068639 7937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:28.088693 7937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40765
I20260924 11:41:28.088711 8013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40765 every 8 connection(s)
I20260924 11:41:28.089870 7937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:28.091955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 7937
I20260924 11:41:28.092187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:28.092808 388 external_mini_cluster.cc:1428] Setting key 5139a686cf67be868c6e0b6c32e60fb8
I20260924 11:41:28.093343 8014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:28.106799 8014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Bootstrap starting.
Sep 24 11:41:28 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:28.110041 8014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:28.111020 8014 log.cc:824] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:28.113519 8014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: No bootstrap required, opened a new log
I20260924 11:41:28.114545 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:28.094580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.0.97.62:40765)
0924 11:41:28.095238 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:28.095248 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:28.095282 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:41:28.097570 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:28.097589 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:28.097930 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:28.098235 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:28.099754 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.101052 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.102833 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.103451 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.103585 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:28.108215 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:28.108224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:28.108236 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:28.108271 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:28.110938 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:28.111617 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:28.111623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:28.111631 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:28.111789 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:28.112174 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:28.112179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:28.112188 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:28.112634 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:28.112901 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:28.113308 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 11:41:28.113669 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
I20260924 11:41:28.120663 8014 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda07ce6a2434a7695e6316874cc0f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40765 } }
I20260924 11:41:28.120985 8014 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:28.121090 8014 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda07ce6a2434a7695e6316874cc0f14, State: Initialized, Role: FOLLOWER
I20260924 11:41:28.121619 8014 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda07ce6a2434a7695e6316874cc0f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40765 } }
I20260924 11:41:28.121822 8014 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:28.121954 8014 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:28.122108 8014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:28.124205 8014 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda07ce6a2434a7695e6316874cc0f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40765 } }
I20260924 11:41:28.124702 8014 leader_election.cc:302] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda07ce6a2434a7695e6316874cc0f14; no voters:
I20260924 11:41:28.125247 8014 leader_election.cc:288] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:28.125337 8019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:28.126142 8019 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [term 1 LEADER]: Becoming Leader. State: Replica: cda07ce6a2434a7695e6316874cc0f14, State: Running, Role: LEADER
I20260924 11:41:28.126596 8019 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda07ce6a2434a7695e6316874cc0f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40765 } }
I20260924 11:41:28.127171 8014 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:28.129426 8020 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda07ce6a2434a7695e6316874cc0f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda07ce6a2434a7695e6316874cc0f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40765 } } }
I20260924 11:41:28.129400 8021 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda07ce6a2434a7695e6316874cc0f14. Latest consensus state: current_term: 1 leader_uuid: "cda07ce6a2434a7695e6316874cc0f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda07ce6a2434a7695e6316874cc0f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40765 } } }
I20260924 11:41:28.129786 8020 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:28.129789 8021 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14 [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:28.134343 8025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:28.139149 8025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:28.146489 8025 catalog_manager.cc:1347] Generated new cluster ID: 0f05d86080b84df080d9ce0e4048e77a
I20260924 11:41:28.146628 8025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:28.168797 8025 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:28.170475 8025 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:28.181883 8025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Generated new TSK 0
I20260924 11:41:28.182886 8025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 11:41:28.234095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40765
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:28.543483 8040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:28.543768 8040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:28.543923 8040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:28.550823 8040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:28.550981 8040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:28.551115 8040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:41:28.551249 8040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:28.560026 8040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:28.561249 8040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:28.562443 8040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:28.574685 8045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:28.574636 8046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:28.574726 8048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:28.575522 8040 server_base.cc:1034] running on GCE node
I20260924 11:41:28.576061 8040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:28.576932 8040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:28.578188 8040 hybrid_clock.cc:648] HybridClock initialized: now 1790250088578134 us; error 43 us; skew 500 ppm
I20260924 11:41:28.578481 8040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:28 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250088, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:28.581684 8040 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 11:41:28.583021 8040 webserver.cc:460] Webserver started at http://127.0.97.1:42077/ using document root <none> and password file <none>
I20260924 11:41:28.583608 8040 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:28.583729 8040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:28.584005 8040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:28.586644 8040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "450c3cbc16d94e76857ab513f25d6423"
format_stamp: "Formatted at 2026-09-24 11:41:28 on dist-test-slave-qhrh"
server_key: "009935c17cf81052f74294055f71d230"
server_key_iv: "0bdfe68608a711f52c88852bb45c54bd"
server_key_version: "encryptionkey@0"
I20260924 11:41:28.587419 8040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "450c3cbc16d94e76857ab513f25d6423"
format_stamp: "Formatted at 2026-09-24 11:41:28 on dist-test-slave-qhrh"
server_key: "009935c17cf81052f74294055f71d230"
server_key_iv: "0bdfe68608a711f52c88852bb45c54bd"
server_key_version: "encryptionkey@0"
I20260924 11:41:28.591952 8040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:41:28.595086 8055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:28.596315 8040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:28.596469 8040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "450c3cbc16d94e76857ab513f25d6423"
format_stamp: "Formatted at 2026-09-24 11:41:28 on dist-test-slave-qhrh"
server_key: "009935c17cf81052f74294055f71d230"
server_key_iv: "0bdfe68608a711f52c88852bb45c54bd"
server_key_version: "encryptionkey@0"
I20260924 11:41:28.596673 8040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:28.613049 8040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:28.616937 8040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:28.617174 8040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:28.620189 8040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:28.620306 8040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 11:41:28.620436 8040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:28.620584 8040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:41:28 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250088, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:28.644469 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:28.620334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35763 (local address 127.0.97.62:40765)
0924 11:41:28.620785 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:28.620794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:28.621828 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 11:41:28.624622 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:28.624626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:28.624691 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:28.624778 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:28.627254 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.628608 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.631779 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.632275 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.632417 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:28.636944 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:28.636948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:28.636951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:28.636981 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:28.640064 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:28.640958 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:28.640966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:28.640970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:28.641081 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:28.641655 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:28.641660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:28.641664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:28.642024 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:28.642181 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:28.644027 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0924 11:41:28.644263 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260924 11:41:28.644840 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:28.620798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.97.1:35763)
0924 11:41:28.621647 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:28.621703 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:28.624333 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:28.625053 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:28.625064 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:28.625957 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:28.626958 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:28.626987 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.628852 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.628860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:28.631627 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:28.631646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.632530 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.632536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:28.632751 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:28.633410 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:28.633443 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:28.636678 (+ 3235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:28.640229 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:28.640237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:28.640250 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:28.640852 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:28.641251 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:28.641257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:28.641262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:28.641524 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:28.642181 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:28.642188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:28.642634 (+ 446us) client_negotiation.cc:769] Sending connection context
0924 11:41:28.642917 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 11:41:28.643313 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":338,"threads_started":1}
I20260924 11:41:28.664703 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:28.649717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.97.1:46291)
0924 11:41:28.650306 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:28.650330 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:28.650448 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:28.650801 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:28.650806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:28.651264 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:28.651501 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:28.651508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.652895 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.652904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:28.654562 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:28.654579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.655847 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.655856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:28.656080 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:28.656879 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:28.656903 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:28.658195 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:28.661317 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:28.661326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:28.661331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:28.661976 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:28.662438 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:28.662442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:28.662446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:28.662608 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:28.663354 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:28.663365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:28.663626 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 11:41:28.663826 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 11:41:28.664279 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":386,"threads_started":1}
I20260924 11:41:28.664822 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:28.650048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46291 (local address 127.0.97.62:40765)
0924 11:41:28.650226 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:28.650232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:28.650419 (+ 187us) server_negotiation.cc:407] Connection header received
0924 11:41:28.650551 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:28.650558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:28.650642 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:28.650739 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:28.651619 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.652711 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.654712 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:28.655460 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:28.655614 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:28.658385 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:28.658406 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:28.658412 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:28.658460 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:28.661180 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:28.662152 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:28.662157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:28.662161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:28.662306 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:28.662737 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:28.662746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:28.662752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:28.663114 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:28.663288 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:28.664338 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0924 11:41:28.664521 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:41:28.667418 8040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35263
I20260924 11:41:28.667438 8175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35263 every 8 connection(s)
I20260924 11:41:28.668846 8040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:28.670542 8176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40765
I20260924 11:41:28.670703 8176 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:28.671202 8176 heartbeater.cc:511] Master 127.0.97.62:40765 requested a full tablet report, sending...
I20260924 11:41:28.673018 7969 ts_manager.cc:194] Registered new tserver with Master: 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263)
I20260924 11:41:28.673117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8040
I20260924 11:41:28.673296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:28.674019 388 external_mini_cluster.cc:1428] Setting key 2ab31feb56d23a78dd68be2f755bf81a
I20260924 11:41:28.674832 7969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35763
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 11:41:28.715201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40765
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:29.016129 8180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:29.016491 8180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:29.016666 8180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:29.025110 8180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:29.025197 8180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:29.025271 8180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:41:29.025344 8180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:29.036906 8180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:29.037948 8180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:29.039175 8180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:29.052382 8185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:29.052295 8186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:29.052376 8188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:29.053448 8180 server_base.cc:1034] running on GCE node
I20260924 11:41:29.053973 8180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:29.054709 8180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:29.055930 8180 hybrid_clock.cc:648] HybridClock initialized: now 1790250089055840 us; error 49 us; skew 500 ppm
I20260924 11:41:29.056259 8180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250089, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:29.059108 8180 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 11:41:29.060458 8180 webserver.cc:460] Webserver started at http://127.0.97.2:36493/ using document root <none> and password file <none>
I20260924 11:41:29.061023 8180 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:29.061100 8180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:29.061345 8180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:29.063858 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
format_stamp: "Formatted at 2026-09-24 11:41:29 on dist-test-slave-qhrh"
server_key: "a08b6adb1777b11c95cb9c440f40b6fb"
server_key_iv: "8edd632c4a4fa5bd3015f00eacc3dbe3"
server_key_version: "encryptionkey@0"
I20260924 11:41:29.064599 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
format_stamp: "Formatted at 2026-09-24 11:41:29 on dist-test-slave-qhrh"
server_key: "a08b6adb1777b11c95cb9c440f40b6fb"
server_key_iv: "8edd632c4a4fa5bd3015f00eacc3dbe3"
server_key_version: "encryptionkey@0"
I20260924 11:41:29.069165 8180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:41:29.072114 8195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:29.073388 8180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:29.073557 8180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
format_stamp: "Formatted at 2026-09-24 11:41:29 on dist-test-slave-qhrh"
server_key: "a08b6adb1777b11c95cb9c440f40b6fb"
server_key_iv: "8edd632c4a4fa5bd3015f00eacc3dbe3"
server_key_version: "encryptionkey@0"
I20260924 11:41:29.073796 8180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:29.094892 8180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:29.099061 8180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:29.099385 8180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:29.102377 8180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:29.102463 8180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:29.102589 8180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:29.102653 8180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250089, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:29.123265 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.102849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37583 (local address 127.0.97.62:40765)
0924 11:41:29.103091 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.103096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.104473 (+ 1377us) server_negotiation.cc:407] Connection header received
0924 11:41:29.106322 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.106329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.106415 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.106542 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:29.108600 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.109632 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.111266 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.111664 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.111793 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.116059 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:29.116067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:29.116073 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:29.116118 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:29.119434 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.120501 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.120508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.120513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.120677 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.121269 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.121278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.121283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.121747 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:29.121956 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.122511 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.122805 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 11:41:29.123581 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.103476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.97.2:37583)
0924 11:41:29.104297 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.104346 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.106080 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.106811 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.106837 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.107550 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:29.108440 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.108458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.109786 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.109792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.111121 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.111132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.111878 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.111886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.112119 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.112843 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:29.112878 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:29.115773 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:29.119595 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.119600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.119609 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.120308 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.120807 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.120811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.120815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.121114 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.121888 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:29.121893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:29.122152 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.122344 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.122632 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":327,"threads_started":1}
I20260924 11:41:29.142928 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.129417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53489 (local address 127.0.97.62:40765)
0924 11:41:29.129636 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.129644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.130198 (+ 554us) server_negotiation.cc:407] Connection header received
0924 11:41:29.130332 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.130340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.130438 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.130561 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:29.131306 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.132607 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.134668 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.135520 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.135687 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.137555 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:29.137563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:29.137570 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:29.137611 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:29.140714 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.141292 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.141299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.141307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.141405 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.141730 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.141735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.141738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.141985 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:29.142122 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.142448 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.142620 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:41:29.143085 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.129452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.97.2:53489)
0924 11:41:29.130095 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.130112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.130231 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.130572 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.130575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.130931 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:29.131184 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.131192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.132765 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.132771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.134531 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.134547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.135701 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.135705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.135825 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.136453 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:29.136469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:29.137370 (+ 901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:29.140833 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.140837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.140841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.141176 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.141516 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.141521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.141524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.141633 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.142092 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:29.142096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:29.142251 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.142372 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.142630 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":371,"threads_started":1}
I20260924 11:41:29.157436 8180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39793
I20260924 11:41:29.157481 8314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39793 every 8 connection(s)
I20260924 11:41:29.158583 8180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:29.160581 8315 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40765
I20260924 11:41:29.160799 8315 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:29.160992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8180
I20260924 11:41:29.161146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:29.161553 8315 heartbeater.cc:511] Master 127.0.97.62:40765 requested a full tablet report, sending...
I20260924 11:41:29.161593 388 external_mini_cluster.cc:1428] Setting key 8aa140f13d5d9b36bfe1b66e256a9cd1
I20260924 11:41:29.163341 7968 ts_manager.cc:194] Registered new tserver with Master: d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793)
I20260924 11:41:29.164575 7968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37583
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 11:41:29.207202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40765
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:29.507010 8319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:29.507266 8319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:29.507424 8319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:29.514493 8319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:29.514582 8319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:29.514715 8319 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:41:29.514832 8319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:29.523742 8319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:29.524855 8319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:29.526175 8319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:29.538884 8324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:29.538909 8327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:29.538854 8325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:29.539883 8319 server_base.cc:1034] running on GCE node
I20260924 11:41:29.540552 8319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:29.541420 8319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:29.542647 8319 hybrid_clock.cc:648] HybridClock initialized: now 1790250089542593 us; error 37 us; skew 500 ppm
I20260924 11:41:29.542922 8319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250089, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:29.545897 8319 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 11:41:29.547238 8319 webserver.cc:460] Webserver started at http://127.0.97.3:35573/ using document root <none> and password file <none>
I20260924 11:41:29.547843 8319 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:29.547924 8319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:29.548226 8319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:29.550825 8319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "227571db98ce448480b581993c242cc9"
format_stamp: "Formatted at 2026-09-24 11:41:29 on dist-test-slave-qhrh"
server_key: "a245e25c94f6c39bb5cf4fd8234e6e2a"
server_key_iv: "6ae6dd040a6e6f93e17ac4b79a2e9096"
server_key_version: "encryptionkey@0"
I20260924 11:41:29.551469 8319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "227571db98ce448480b581993c242cc9"
format_stamp: "Formatted at 2026-09-24 11:41:29 on dist-test-slave-qhrh"
server_key: "a245e25c94f6c39bb5cf4fd8234e6e2a"
server_key_iv: "6ae6dd040a6e6f93e17ac4b79a2e9096"
server_key_version: "encryptionkey@0"
I20260924 11:41:29.555794 8319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 11:41:29.558877 8334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:29.560092 8319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:29.560272 8319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "227571db98ce448480b581993c242cc9"
format_stamp: "Formatted at 2026-09-24 11:41:29 on dist-test-slave-qhrh"
server_key: "a245e25c94f6c39bb5cf4fd8234e6e2a"
server_key_iv: "6ae6dd040a6e6f93e17ac4b79a2e9096"
server_key_version: "encryptionkey@0"
I20260924 11:41:29.560483 8319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:29.573769 8319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:29.577610 8319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:29.577875 8319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:29.580117 8319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:29.580284 8319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:29.580472 8319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:29.580564 8319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250089, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:29.604984 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.580960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51303 (local address 127.0.97.62:40765)
0924 11:41:29.581153 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.581159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.582566 (+ 1407us) server_negotiation.cc:407] Connection header received
0924 11:41:29.585075 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.585081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.585187 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.585316 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:29.587570 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.588865 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.591828 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.592568 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.592692 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.596713 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:29.596721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:29.596734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:29.596782 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:29.600371 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.601511 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.601519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.601523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.601681 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.602240 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.602247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.602253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.602616 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:29.602761 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.604361 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.604604 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:41:29.605469 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.581494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.97.3:51303)
0924 11:41:29.582376 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.582426 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.584763 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.585591 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.585606 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.586451 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:29.587320 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.587372 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.589073 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.589084 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.591611 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.591627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.592816 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.592840 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.593008 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.593575 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:29.593597 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:29.596418 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:29.600546 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.600557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.600574 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.601318 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.601824 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.601829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.601833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.602110 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.602784 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:29.602799 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:29.603161 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.603497 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.604048 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":389,"threads_started":1}
I20260924 11:41:29.625655 8319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42313
I20260924 11:41:29.625756 8453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42313 every 8 connection(s)
I20260924 11:41:29.627177 8319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:29.629302 8454 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40765
I20260924 11:41:29.629492 8454 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:29.630137 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.614838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.97.3:54909)
0924 11:41:29.615437 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.615458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.615603 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.615920 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.615925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.616462 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:29.616943 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.616984 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.618038 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.618044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.620140 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.620191 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.621483 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.621496 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.621735 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.622398 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:29.622428 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:29.623596 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:29.627426 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.627431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.627435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.628031 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.628507 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:29.628514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:29.628519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:29.628667 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:29.629226 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:29.629230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:29.629427 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.629624 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.629826 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":352,"threads_started":1}
I20260924 11:41:29.630306 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.614722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54909 (local address 127.0.97.62:40765)
0924 11:41:29.614858 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.614861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.615543 (+ 682us) server_negotiation.cc:407] Connection header received
0924 11:41:29.615735 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.615746 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.615800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.615900 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:29.617146 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.617881 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.620409 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.620979 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.621126 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.623774 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:29.623779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:29.623783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:29.623819 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:29.627285 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.628208 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.628213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.628216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.628360 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.628782 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.628788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.628791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.629112 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:29.629243 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.629759 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.629964 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:41:29.630398 8454 heartbeater.cc:511] Master 127.0.97.62:40765 requested a full tablet report, sending...
I20260924 11:41:29.631652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8319
I20260924 11:41:29.631711 7968 ts_manager.cc:194] Registered new tserver with Master: 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:29.631836 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:29.632313 388 external_mini_cluster.cc:1428] Setting key 886fc876bedce9b19fe565f209644400
I20260924 11:41:29.632699 7968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51303
I20260924 11:41:29.635485 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:29.677448 8176 heartbeater.cc:503] Master 127.0.97.62:40765 was elected leader, sending a full tablet report...
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:41:29.679783 8063 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:29.669681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.97.62:40765)
0924 11:41:29.669877 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.669882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.670129 (+ 247us) server_negotiation.cc:407] Connection header received
0924 11:41:29.670284 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.670288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.670354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.670435 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:29.671954 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.672675 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.673833 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.674194 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.674276 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.679598 (+ 5322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56504: BlockingRecv error: recv got EOF from 127.0.0.1:56504 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 11:41:29 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:29.693751 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.682670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.97.62:40765)
0924 11:41:29.682819 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.682823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.683206 (+ 383us) server_negotiation.cc:407] Connection header received
0924 11:41:29.683303 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.683306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.683363 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.683432 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:29.684409 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.684983 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.686162 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.686535 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.686647 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.689798 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:29.689803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:29.689806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:29.689830 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:29.691943 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.692511 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.692514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.692516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.692619 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:29.692893 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:29.692896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:29.692898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:29.693108 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:29.693293 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.693438 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.693567 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:41:29.697148 7968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:29.703802 7968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:29.736334 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.726907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.0.97.2:39793)
0924 11:41:29.729089 (+ 2182us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.729099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.729122 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:41:29.729219 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.729225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.729440 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.729644 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:29.730190 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.731462 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.734000 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.735108 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.735234 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.735416 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.735851 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.736003 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2041,"thread_start_us":1911,"threads_started":1}
I20260924 11:41:29.736408 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.714466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.97.1:35263)
0924 11:41:29.725560 (+ 11094us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.725571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.727255 (+ 1684us) server_negotiation.cc:407] Connection header received
0924 11:41:29.727343 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.727349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.727531 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.727683 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:29.729912 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.730962 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.733599 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.734951 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.735065 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.735247 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.735943 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.736103 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10488,"spinlock_wait_cycles":452480,"thread_start_us":10522,"threads_started":1}
I20260924 11:41:29.736444 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.714291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35263 (local address 127.0.0.1:43656)
0924 11:41:29.726993 (+ 12702us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.727011 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.727124 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.727905 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.727912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.727949 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:29.729028 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.729039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.731096 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.731101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.733445 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.733459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.735395 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.735401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.735694 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.735787 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.735866 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.735978 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11591,"mutex_wait_us":120,"spinlock_wait_cycles":384,"thread_start_us":11968,"threads_started":1}
I20260924 11:41:29.736858 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.727171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39793 (local address 127.0.0.1:34808)
0924 11:41:29.728512 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.728537 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.728634 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.729632 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.729635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.729654 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:29.729991 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.730003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.731619 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.731624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.733869 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.733898 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.735554 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.735558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.735683 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.735715 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.735803 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.735976 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"mutex_wait_us":22,"thread_start_us":1064,"threads_started":1}
I20260924 11:41:29.737229 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.714478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42313 (local address 127.0.0.1:52628)
0924 11:41:29.726582 (+ 12104us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.726618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.726785 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.727919 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.727923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.727956 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:29.728464 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.728476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.730885 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.730892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.733298 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.733315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.735424 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.735429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.735706 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.736515 (+ 809us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.736625 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.736743 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11923,"mutex_wait_us":912,"spinlock_wait_cycles":4480,"thread_start_us":11268,"threads_started":1}
I20260924 11:41:29.737393 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.715043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.0.97.3:42313)
0924 11:41:29.727097 (+ 12054us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.727106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.727137 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:41:29.727229 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.727236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.727461 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.727683 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:29.729225 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.730746 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.733460 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.734949 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.735065 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.735273 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.736945 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.737080 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11893,"thread_start_us":11654,"threads_started":1}
I20260924 11:41:29.739763 8106 tablet_service.cc:1467] Processing CreateTablet for tablet e8b8588ee380408b8c6dd40e7f81cbc3 (DEFAULT_TABLE table=test-table [id=4fb727477c91414ba66505fe3922be61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:29.740022 8384 tablet_service.cc:1467] Processing CreateTablet for tablet e8b8588ee380408b8c6dd40e7f81cbc3 (DEFAULT_TABLE table=test-table [id=4fb727477c91414ba66505fe3922be61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:29.740965 8106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b8588ee380408b8c6dd40e7f81cbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:29.741313 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b8588ee380408b8c6dd40e7f81cbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:29.743045 8245 tablet_service.cc:1467] Processing CreateTablet for tablet e8b8588ee380408b8c6dd40e7f81cbc3 (DEFAULT_TABLE table=test-table [id=4fb727477c91414ba66505fe3922be61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:29.744071 8245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b8588ee380408b8c6dd40e7f81cbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:29.752470 8478 tablet_bootstrap.cc:492] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Bootstrap starting.
I20260924 11:41:29.755004 8478 tablet_bootstrap.cc:654] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:29.755898 8478 log.cc:824] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:29.756968 8479 tablet_bootstrap.cc:492] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Bootstrap starting.
I20260924 11:41:29.759195 8481 tablet_bootstrap.cc:492] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Bootstrap starting.
I20260924 11:41:29.760073 8479 tablet_bootstrap.cc:654] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:29.761333 8479 log.cc:824] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:29.761965 8481 tablet_bootstrap.cc:654] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:29.763609 8481 log.cc:824] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:29.764581 8478 tablet_bootstrap.cc:492] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: No bootstrap required, opened a new log
I20260924 11:41:29.764851 8478 ts_tablet_manager.cc:1397] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260924 11:41:29.766220 8479 tablet_bootstrap.cc:492] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: No bootstrap required, opened a new log
I20260924 11:41:29.766449 8479 ts_tablet_manager.cc:1397] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 11:41:29.769831 8481 tablet_bootstrap.cc:492] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: No bootstrap required, opened a new log
I20260924 11:41:29.770080 8481 ts_tablet_manager.cc:1397] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260924 11:41:29.775830 8478 raft_consensus.cc:348] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.776253 8478 raft_consensus.cc:374] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:29.776342 8478 raft_consensus.cc:729] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227571db98ce448480b581993c242cc9, State: Initialized, Role: FOLLOWER
I20260924 11:41:29.776393 8479 raft_consensus.cc:348] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.776373 8481 raft_consensus.cc:348] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.776728 8479 raft_consensus.cc:374] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:29.776741 8481 raft_consensus.cc:374] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:29.776811 8479 raft_consensus.cc:729] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Initialized, Role: FOLLOWER
I20260924 11:41:29.776834 8481 raft_consensus.cc:729] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a8e15a7dda4a0c8bd0b39696d3a0c8, State: Initialized, Role: FOLLOWER
I20260924 11:41:29.777313 8479 consensus_queue.cc:260] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.777441 8481 consensus_queue.cc:260] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.777926 8478 consensus_queue.cc:260] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.779284 8315 heartbeater.cc:503] Master 127.0.97.62:40765 was elected leader, sending a full tablet report...
I20260924 11:41:29.779551 8481 ts_tablet_manager.cc:1428] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent starting tablet: real 0.009s user 0.004s sys 0.006s
I20260924 11:41:29.779593 8479 ts_tablet_manager.cc:1428] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260924 11:41:29.780328 8454 heartbeater.cc:503] Master 127.0.97.62:40765 was elected leader, sending a full tablet report...
I20260924 11:41:29.781791 8478 ts_tablet_manager.cc:1428] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260924 11:41:29.878919 8455 tablet.cc:2369] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:29.891844 8485 raft_consensus.cc:482] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:29.892119 8485 raft_consensus.cc:504] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.893679 8485 leader_election.cc:288] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263), 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:29.900245 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.894060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42827 (local address 127.0.97.1:35263)
0924 11:41:29.894285 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.894294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.895115 (+ 821us) server_negotiation.cc:407] Connection header received
0924 11:41:29.895306 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.895317 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.895383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.895475 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:29.896120 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.896842 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.898560 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.899322 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.899462 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.899558 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.899895 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.900001 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:41:29.900306 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.894140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56193 (local address 127.0.97.3:42313)
0924 11:41:29.894356 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.894364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.895070 (+ 706us) server_negotiation.cc:407] Connection header received
0924 11:41:29.895290 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.895294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.895350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.895442 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:29.896059 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.896799 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.898585 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.899356 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.899486 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.899581 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.899944 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.900051 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:41:29.900488 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.894086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35263 (local address 127.0.97.2:42827)
0924 11:41:29.895007 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.895020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.895176 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.895487 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.895492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.895512 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:29.895958 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.895967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.896966 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.896975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.898415 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.898426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.899528 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.899533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.899693 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.899766 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.899863 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.899972 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":39,"thread_start_us":389,"threads_started":1}
I20260924 11:41:29.900542 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.894059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42313 (local address 127.0.97.2:56193)
0924 11:41:29.894925 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:29.894957 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:29.895132 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:29.895465 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:29.895469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:29.895500 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:29.895919 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.895928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.896920 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.896927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.898425 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:29.898435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.899557 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.899563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:29.899776 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.899796 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:41:29.899891 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:41:29.899994 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":80,"thread_start_us":528,"threads_started":1}
I20260924 11:41:29.901412 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8588ee380408b8c6dd40e7f81cbc3" candidate_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3cbc16d94e76857ab513f25d6423" is_pre_election: true
I20260924 11:41:29.901410 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8588ee380408b8c6dd40e7f81cbc3" candidate_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227571db98ce448480b581993c242cc9" is_pre_election: true
I20260924 11:41:29.901849 8131 raft_consensus.cc:2454] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a8e15a7dda4a0c8bd0b39696d3a0c8 in term 0.
I20260924 11:41:29.901865 8404 raft_consensus.cc:2454] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a8e15a7dda4a0c8bd0b39696d3a0c8 in term 0.
I20260924 11:41:29.902432 8197 leader_election.cc:302] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, d9a8e15a7dda4a0c8bd0b39696d3a0c8; no voters:
I20260924 11:41:29.902808 8485 raft_consensus.cc:2781] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:29.902915 8485 raft_consensus.cc:482] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:29.903016 8485 raft_consensus.cc:3037] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:29.905306 8485 raft_consensus.cc:504] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
I20260924 11:41:29.905915 8485 leader_election.cc:288] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [CANDIDATE]: Term 1 election: Requested vote from peers 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263), 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:29.906200 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8588ee380408b8c6dd40e7f81cbc3" candidate_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3cbc16d94e76857ab513f25d6423"
I20260924 11:41:29.906373 8131 raft_consensus.cc:3037] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:29.906378 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b8588ee380408b8c6dd40e7f81cbc3" candidate_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227571db98ce448480b581993c242cc9"
I20260924 11:41:29.906548 8404 raft_consensus.cc:3037] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:29.908691 8131 raft_consensus.cc:2454] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a8e15a7dda4a0c8bd0b39696d3a0c8 in term 1.
I20260924 11:41:29.909029 8404 raft_consensus.cc:2454] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a8e15a7dda4a0c8bd0b39696d3a0c8 in term 1.
I20260924 11:41:29.909117 8198 leader_election.cc:302] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450c3cbc16d94e76857ab513f25d6423, d9a8e15a7dda4a0c8bd0b39696d3a0c8; no voters:
I20260924 11:41:29.909399 8485 raft_consensus.cc:2781] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:29.909793 8485 raft_consensus.cc:686] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 LEADER]: Becoming Leader. State: Replica: d9a8e15a7dda4a0c8bd0b39696d3a0c8, State: Running, Role: LEADER
I20260924 11:41:29.910218 8485 consensus_queue.cc:237] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } }
W20260924 11:41:29.910594 8316 tablet.cc:2369] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:29.914803 7969 catalog_manager.cc:5495] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } health_report { overall_health: HEALTHY } } }
W20260924 11:41:29.921068 8177 tablet.cc:2369] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:41:29.971900 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:29.966937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.0.97.2:39793)
0924 11:41:29.967116 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:29.967120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:29.967138 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:41:29.967373 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:29.967377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:29.967449 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:29.967566 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:29.967870 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.968633 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.969855 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:29.970229 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:29.970314 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:29.970682 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:29.971155 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:29.971279 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:29.971550 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 11:41:29.971609 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:41:29.979329 8131 raft_consensus.cc:1261] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Refusing update from remote peer d9a8e15a7dda4a0c8bd0b39696d3a0c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:29.979379 8404 raft_consensus.cc:1261] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Refusing update from remote peer d9a8e15a7dda4a0c8bd0b39696d3a0c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:29.980234 8489 consensus_queue.cc:1035] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:29.980650 8485 consensus_queue.cc:1035] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:29.991864 8494 mvcc.cc:205] Tried to move back new op lower bound from 7332864368546443264 to 7332864368276795392. Current Snapshot: MvccSnapshot[applied={T|T < 7332864368546443264}]
I20260924 11:41:30.002156 7969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:30.002430 7969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:30.005741 7969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Sending DeleteTablet for 3 replicas of tablet e8b8588ee380408b8c6dd40e7f81cbc3
I20260924 11:41:30.006771 8245 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b8588ee380408b8c6dd40e7f81cbc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34808
I20260924 11:41:30.006822 8384 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b8588ee380408b8c6dd40e7f81cbc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52628
I20260924 11:41:30.007666 8501 tablet_replica.cc:330] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: stopping tablet replica
I20260924 11:41:30.007663 8106 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b8588ee380408b8c6dd40e7f81cbc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43656
I20260924 11:41:30.008260 8501 raft_consensus.cc:2229] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:30.008967 8501 raft_consensus.cc:2258] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.011175 8501 ts_tablet_manager.cc:1905] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.013339 7969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:30.013931 7969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:30.019585 8503 tablet_replica.cc:330] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: stopping tablet replica
I20260924 11:41:30.019982 8503 raft_consensus.cc:2229] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:30.020447 8503 raft_consensus.cc:2258] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.021202 8501 ts_tablet_manager.cc:1918] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:30.021422 8501 log.cc:1196] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/e8b8588ee380408b8c6dd40e7f81cbc3
I20260924 11:41:30.022007 8501 ts_tablet_manager.cc:1939] T e8b8588ee380408b8c6dd40e7f81cbc3 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting consensus metadata
I20260924 11:41:30.022557 8503 ts_tablet_manager.cc:1905] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.024330 7955 catalog_manager.cc:4841] TS d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793): tablet e8b8588ee380408b8c6dd40e7f81cbc3 (table test-table [id=4fb727477c91414ba66505fe3922be61]) successfully deleted
I20260924 11:41:30.024348 8383 tablet_service.cc:1467] Processing CreateTablet for tablet 9dcc2904748942e7bafbe0076cb5741c (DEFAULT_TABLE table=test-table [id=bd4b3398b6e74a0ca6e5597a046d9383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.024550 8245 tablet_service.cc:1467] Processing CreateTablet for tablet 9dcc2904748942e7bafbe0076cb5741c (DEFAULT_TABLE table=test-table [id=bd4b3398b6e74a0ca6e5597a046d9383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.024863 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dcc2904748942e7bafbe0076cb5741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.024981 8245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dcc2904748942e7bafbe0076cb5741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.029367 8503 ts_tablet_manager.cc:1918] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:30.029505 8503 log.cc:1196] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/e8b8588ee380408b8c6dd40e7f81cbc3
I20260924 11:41:30.030014 8503 ts_tablet_manager.cc:1939] T e8b8588ee380408b8c6dd40e7f81cbc3 P 450c3cbc16d94e76857ab513f25d6423: Deleting consensus metadata
I20260924 11:41:30.031454 8481 tablet_bootstrap.cc:492] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Bootstrap starting.
I20260924 11:41:30.031490 8478 tablet_bootstrap.cc:492] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Bootstrap starting.
I20260924 11:41:30.031912 8106 tablet_service.cc:1467] Processing CreateTablet for tablet 9dcc2904748942e7bafbe0076cb5741c (DEFAULT_TABLE table=test-table [id=bd4b3398b6e74a0ca6e5597a046d9383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.032310 8106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dcc2904748942e7bafbe0076cb5741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.033027 8478 tablet_bootstrap.cc:654] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.033052 8481 tablet_bootstrap.cc:654] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.033931 7955 catalog_manager.cc:4841] TS 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263): tablet e8b8588ee380408b8c6dd40e7f81cbc3 (table test-table [id=4fb727477c91414ba66505fe3922be61]) successfully deleted
I20260924 11:41:30.039301 8479 tablet_bootstrap.cc:492] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Bootstrap starting.
I20260924 11:41:30.041334 8479 tablet_bootstrap.cc:654] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.043588 8478 tablet_bootstrap.cc:492] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: No bootstrap required, opened a new log
I20260924 11:41:30.043820 8478 ts_tablet_manager.cc:1397] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260924 11:41:30.044853 8478 raft_consensus.cc:348] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.045236 8478 raft_consensus.cc:374] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.045383 8478 raft_consensus.cc:729] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227571db98ce448480b581993c242cc9, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.045724 8478 consensus_queue.cc:260] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.046458 8478 ts_tablet_manager.cc:1428] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:30.047933 8481 tablet_bootstrap.cc:492] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: No bootstrap required, opened a new log
I20260924 11:41:30.048074 8481 ts_tablet_manager.cc:1397] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260924 11:41:30.048870 8481 raft_consensus.cc:348] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.049052 8481 raft_consensus.cc:374] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.049150 8481 raft_consensus.cc:729] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a8e15a7dda4a0c8bd0b39696d3a0c8, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.049387 8481 consensus_queue.cc:260] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.050020 8481 ts_tablet_manager.cc:1428] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:30.050304 8502 tablet_replica.cc:330] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: stopping tablet replica
I20260924 11:41:30.051225 8502 raft_consensus.cc:2229] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:30.051733 8502 raft_consensus.cc:2258] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.054289 8502 ts_tablet_manager.cc:1905] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.060038 8479 tablet_bootstrap.cc:492] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: No bootstrap required, opened a new log
I20260924 11:41:30.060210 8479 ts_tablet_manager.cc:1397] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20260924 11:41:30.061024 8479 raft_consensus.cc:348] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.061264 8479 raft_consensus.cc:374] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.061371 8479 raft_consensus.cc:729] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.061480 8502 ts_tablet_manager.cc:1918] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:30.061622 8502 log.cc:1196] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/e8b8588ee380408b8c6dd40e7f81cbc3
I20260924 11:41:30.061699 8479 consensus_queue.cc:260] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.062114 8502 ts_tablet_manager.cc:1939] T e8b8588ee380408b8c6dd40e7f81cbc3 P 227571db98ce448480b581993c242cc9: Deleting consensus metadata
I20260924 11:41:30.062469 8479 ts_tablet_manager.cc:1428] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:30.063868 7954 catalog_manager.cc:4841] TS 227571db98ce448480b581993c242cc9 (127.0.97.3:42313): tablet e8b8588ee380408b8c6dd40e7f81cbc3 (table test-table [id=4fb727477c91414ba66505fe3922be61]) successfully deleted
I20260924 11:41:30.078970 8484 raft_consensus.cc:482] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:30.079154 8484 raft_consensus.cc:504] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.080674 8484 leader_election.cc:288] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793), 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:30.088769 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.080990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39793 (local address 127.0.97.1:47877)
0924 11:41:30.082791 (+ 1801us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.082804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.082951 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.083295 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.083300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.083324 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:30.083631 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.083641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.084825 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.084831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.086287 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.086298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.087903 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.087910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.088109 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.088184 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.088269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.088395 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"mutex_wait_us":657,"thread_start_us":850,"threads_started":1}
I20260924 11:41:30.088884 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.081430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58811 (local address 127.0.97.3:42313)
0924 11:41:30.081607 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.081612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.081948 (+ 336us) server_negotiation.cc:407] Connection header received
0924 11:41:30.082200 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.082205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.082275 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.082337 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:30.083188 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.084101 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.086616 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.087795 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.087895 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.087971 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.088493 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.088617 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:41:30.089493 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.080985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42313 (local address 127.0.97.1:58811)
0924 11:41:30.081858 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.081877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.082044 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.082556 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.082560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.082589 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:30.082980 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.082991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.084239 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.084244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.086470 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.086483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.088124 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.088130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.088320 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.088352 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.089019 (+ 667us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.089138 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":39,"thread_start_us":532,"threads_started":1}
I20260924 11:41:30.090049 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.081126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47877 (local address 127.0.97.2:39793)
0924 11:41:30.081306 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.081312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.082892 (+ 1580us) server_negotiation.cc:407] Connection header received
0924 11:41:30.083092 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.083096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.083175 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.083263 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:30.083800 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.084709 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.086423 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.087669 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.087822 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.087928 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.089472 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.089769 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:41:30.090848 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dcc2904748942e7bafbe0076cb5741c" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227571db98ce448480b581993c242cc9" is_pre_election: true
I20260924 11:41:30.091076 8404 raft_consensus.cc:2454] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 0.
I20260924 11:41:30.090996 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dcc2904748942e7bafbe0076cb5741c" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" is_pre_election: true
I20260924 11:41:30.091428 8270 raft_consensus.cc:2454] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 0.
I20260924 11:41:30.091504 8057 leader_election.cc:302] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, 450c3cbc16d94e76857ab513f25d6423; no voters:
I20260924 11:41:30.091861 8484 raft_consensus.cc:2781] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:30.091967 8484 raft_consensus.cc:482] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:30.092061 8484 raft_consensus.cc:3037] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.095005 8484 raft_consensus.cc:504] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.095981 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dcc2904748942e7bafbe0076cb5741c" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227571db98ce448480b581993c242cc9"
I20260924 11:41:30.095985 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dcc2904748942e7bafbe0076cb5741c" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
I20260924 11:41:30.096161 8404 raft_consensus.cc:3037] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.096191 8270 raft_consensus.cc:3037] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.098872 8404 raft_consensus.cc:2454] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 1.
I20260924 11:41:30.099262 8057 leader_election.cc:302] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, 450c3cbc16d94e76857ab513f25d6423; no voters:
I20260924 11:41:30.099335 8270 raft_consensus.cc:2454] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 1.
I20260924 11:41:30.099542 8484 leader_election.cc:288] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 election: Requested vote from peers d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793), 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:30.099671 8484 raft_consensus.cc:2781] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:30.099776 8484 raft_consensus.cc:686] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 1 LEADER]: Becoming Leader. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Running, Role: LEADER
I20260924 11:41:30.100118 8484 consensus_queue.cc:237] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.105485 7967 catalog_manager.cc:5495] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "450c3cbc16d94e76857ab513f25d6423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:41:30 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250088, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:30.148455 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.134481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40765 (local address 127.0.0.1:56512)
0924 11:41:30.135061 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.135075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.135181 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.135485 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.135488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.136136 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:30.136447 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.136456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.137834 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.137837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.138776 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.138785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.139671 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.139675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.139774 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.140427 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:30.140448 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:30.142559 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:30.145872 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:30.145881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:30.145886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:30.146375 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:30.146757 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:30.146760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:30.146762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:30.146928 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:30.147577 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:30.147597 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:30.147745 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.147890 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.148064 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":317,"threads_started":1}
I20260924 11:41:30.148669 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.134437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.97.62:40765)
0924 11:41:30.134648 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.134654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.135148 (+ 494us) server_negotiation.cc:407] Connection header received
0924 11:41:30.135282 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.135287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.135357 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.135463 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:30.136579 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.137719 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.138903 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.139516 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.139638 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.142716 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:30.142722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:30.142727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:30.142758 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:30.145721 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:30.146500 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:30.146506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:30.146510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:30.146638 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:30.147071 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:30.147076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:30.147080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:30.147434 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:30.147572 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.148000 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.148229 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 11:41:30.153487 7967 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56512:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:41:30.154616 7967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:30.161927 8384 tablet_service.cc:1467] Processing CreateTablet for tablet 883d82be64e544f1bbcb13c94482cefa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9db30be0e6c54b3b94c6014a223aadf1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:41:30.162078 8106 tablet_service.cc:1467] Processing CreateTablet for tablet 883d82be64e544f1bbcb13c94482cefa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9db30be0e6c54b3b94c6014a223aadf1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:41:30.162403 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883d82be64e544f1bbcb13c94482cefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.162592 8106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883d82be64e544f1bbcb13c94482cefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.168831 8245 tablet_service.cc:1467] Processing CreateTablet for tablet 883d82be64e544f1bbcb13c94482cefa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9db30be0e6c54b3b94c6014a223aadf1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:41:30.169047 8478 tablet_bootstrap.cc:492] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: Bootstrap starting.
I20260924 11:41:30.169360 8245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883d82be64e544f1bbcb13c94482cefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.170610 8478 tablet_bootstrap.cc:654] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.177757 8479 tablet_bootstrap.cc:492] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: Bootstrap starting.
I20260924 11:41:30.177783 8478 tablet_bootstrap.cc:492] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: No bootstrap required, opened a new log
I20260924 11:41:30.177912 8478 ts_tablet_manager.cc:1397] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:41:30.178705 8478 raft_consensus.cc:348] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.178923 8478 raft_consensus.cc:374] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.178997 8478 raft_consensus.cc:729] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227571db98ce448480b581993c242cc9, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.179346 8479 tablet_bootstrap.cc:654] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.179481 8481 tablet_bootstrap.cc:492] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Bootstrap starting.
I20260924 11:41:30.180094 8478 consensus_queue.cc:260] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.180775 8478 ts_tablet_manager.cc:1428] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 11:41:30.181164 8481 tablet_bootstrap.cc:654] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.182282 8486 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.182598 8486 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.189503 8481 tablet_bootstrap.cc:492] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: No bootstrap required, opened a new log
I20260924 11:41:30.189680 8481 ts_tablet_manager.cc:1397] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260924 11:41:30.190461 8481 raft_consensus.cc:348] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.190728 8481 raft_consensus.cc:374] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.190827 8481 raft_consensus.cc:729] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a8e15a7dda4a0c8bd0b39696d3a0c8, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.191107 8481 consensus_queue.cc:260] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.191577 8489 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.191798 8481 ts_tablet_manager.cc:1428] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 11:41:30.191885 8489 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.192817 8479 tablet_bootstrap.cc:492] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: No bootstrap required, opened a new log
I20260924 11:41:30.192952 8479 ts_tablet_manager.cc:1397] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260924 11:41:30.193661 8479 raft_consensus.cc:348] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.193899 8479 raft_consensus.cc:374] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.193979 8479 raft_consensus.cc:729] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.198416 8479 consensus_queue.cc:260] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.198931 8484 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.199110 8479 ts_tablet_manager.cc:1428] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260924 11:41:30.199206 8484 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.256623 8486 raft_consensus.cc:482] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:30.256830 8486 raft_consensus.cc:504] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.258106 8486 leader_election.cc:288] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263), d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793)
I20260924 11:41:30.265324 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.258587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39513 (local address 127.0.97.1:35263)
0924 11:41:30.258801 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.258807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.259059 (+ 252us) server_negotiation.cc:407] Connection header received
0924 11:41:30.259336 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.259341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.259420 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.259486 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:30.260123 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.261117 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.262986 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.264368 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.264555 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.264685 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.264995 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.265132 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:41:30.265954 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.259076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39793 (local address 127.0.97.3:52513)
0924 11:41:30.259760 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.259774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.259870 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.260278 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.260282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.260305 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:30.260582 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.260594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.262006 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.262011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.263522 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.263532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.265035 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.265043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.265242 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.265282 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.265392 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.265525 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":374,"threads_started":1}
I20260924 11:41:30.266150 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.258391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35263 (local address 127.0.97.3:39513)
0924 11:41:30.258930 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.258948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.259158 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.259720 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.259724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.259751 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:41:30.259993 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.260002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.261241 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.261246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.262838 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.262850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.264580 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.264586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.264804 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.264838 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.265566 (+ 728us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.265719 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":1,"thread_start_us":306,"threads_started":1}
I20260924 11:41:30.265998 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.259195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52513 (local address 127.0.97.2:39793)
0924 11:41:30.259357 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.259362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.259868 (+ 506us) server_negotiation.cc:407] Connection header received
0924 11:41:30.260014 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.260022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.260127 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.260271 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:41:30.260757 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.261885 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.263661 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.264824 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.264960 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.265058 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.265602 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.265739 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:41:30.266723 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d82be64e544f1bbcb13c94482cefa" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3cbc16d94e76857ab513f25d6423" is_pre_election: true
I20260924 11:41:30.266955 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d82be64e544f1bbcb13c94482cefa" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" is_pre_election: true
I20260924 11:41:30.267036 8131 raft_consensus.cc:2454] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 0.
I20260924 11:41:30.267242 8270 raft_consensus.cc:2454] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 0.
I20260924 11:41:30.267479 8337 leader_election.cc:302] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, 450c3cbc16d94e76857ab513f25d6423; no voters:
I20260924 11:41:30.267860 8486 raft_consensus.cc:2781] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:30.267982 8486 raft_consensus.cc:482] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:30.268083 8486 raft_consensus.cc:3037] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.270131 8486 raft_consensus.cc:504] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.271039 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d82be64e544f1bbcb13c94482cefa" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3cbc16d94e76857ab513f25d6423"
I20260924 11:41:30.270694 8486 leader_election.cc:288] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 election: Requested vote from peers 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263), d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793)
I20260924 11:41:30.271154 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d82be64e544f1bbcb13c94482cefa" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
I20260924 11:41:30.271221 8131 raft_consensus.cc:3037] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.271354 8270 raft_consensus.cc:3037] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.273568 8270 raft_consensus.cc:2454] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 1.
I20260924 11:41:30.273932 8337 leader_election.cc:302] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, d9a8e15a7dda4a0c8bd0b39696d3a0c8; no voters:
I20260924 11:41:30.274144 8131 raft_consensus.cc:2454] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 1.
I20260924 11:41:30.274199 8486 raft_consensus.cc:2781] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:30.276382 8486 raft_consensus.cc:686] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 227571db98ce448480b581993c242cc9, State: Running, Role: LEADER
I20260924 11:41:30.276777 8486 consensus_queue.cc:237] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.278874 8522 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: TxnStatusTablet state changed. Reason: New leader 227571db98ce448480b581993c242cc9. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.279276 8522 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:41:30.279937 8524 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:41:30.280130 8524 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:41:30.280586 7967 catalog_manager.cc:5495] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 227571db98ce448480b581993c242cc9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } health_report { overall_health: HEALTHY } } }
I20260924 11:41:30.307117 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.301628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.97.3:42313)
0924 11:41:30.301824 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.301830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.301959 (+ 129us) server_negotiation.cc:407] Connection header received
0924 11:41:30.302215 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.302219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.302299 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.302427 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:30.302747 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.303485 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.304827 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.305344 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.305506 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.305891 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.306450 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.306618 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.306700 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.306769 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:41:30.307161 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.301590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42313 (local address 127.0.0.1:52636)
0924 11:41:30.301873 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.301895 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.302040 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.302443 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.302447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.302458 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:41:30.302630 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.302637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.303606 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.303614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.304684 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.304695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.305514 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.305520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.305669 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.305715 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.306560 (+ 845us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.306586 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.306673 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.306808 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260924 11:41:30.310770 8131 raft_consensus.cc:1261] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Refusing update from remote peer 227571db98ce448480b581993c242cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:30.311069 8270 raft_consensus.cc:1261] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Refusing update from remote peer 227571db98ce448480b581993c242cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:30.311764 8486 consensus_queue.cc:1035] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:30.312052 8522 consensus_queue.cc:1035] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:30.324820 8489 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: TxnStatusTablet state changed. Reason: New leader 227571db98ce448480b581993c242cc9. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.325254 8489 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.325133 8484 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: TxnStatusTablet state changed. Reason: New leader 227571db98ce448480b581993c242cc9. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.325429 8484 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.329303 8486 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.329569 8486 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:41:30.329754 8493 mvcc.cc:205] Tried to move back new op lower bound from 7332864369906675712 to 7332864369777500160. Current Snapshot: MvccSnapshot[applied={T|T < 7332864369906675712 or (T in {7332864369906675712})}]
I20260924 11:41:30.331414 8484 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.331699 8484 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 450c3cbc16d94e76857ab513f25d6423: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.336951 8522 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.337198 8522 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P 227571db98ce448480b581993c242cc9: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:41:30.339053 8492 mvcc.cc:205] Tried to move back new op lower bound from 7332864369906675712 to 7332864369777500160. Current Snapshot: MvccSnapshot[applied={T|T < 7332864369906675712}]
I20260924 11:41:30.340309 8489 tablet_replica.cc:439] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } } }
I20260924 11:41:30.340562 8489 tablet_replica.cc:442] T 883d82be64e544f1bbcb13c94482cefa P d9a8e15a7dda4a0c8bd0b39696d3a0c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:41:30.343474 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.335967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.0.97.1:35263)
0924 11:41:30.336111 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.336116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.336432 (+ 316us) server_negotiation.cc:407] Connection header received
0924 11:41:30.336515 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.336521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.336625 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.336771 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:30.337352 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.338417 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.340415 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.340995 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.341108 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.341759 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.342459 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.342576 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.343090 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.343153 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:41:30.367161 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.360824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42105 (local address 127.0.97.3:42313)
0924 11:41:30.360980 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.360986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.361185 (+ 199us) server_negotiation.cc:407] Connection header received
0924 11:41:30.361248 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.361253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.361320 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.361393 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:30.361965 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.362918 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.365064 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.365634 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.365728 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.366332 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.366458 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.366523 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.366733 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.366803 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:41:30.367995 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.360410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42313 (local address 127.0.97.1:42105)
0924 11:41:30.360964 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.360983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.361117 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.361512 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.361516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.361526 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:41:30.361824 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.361832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.363155 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.363160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.364908 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.364921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.365946 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.365951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.366103 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.366157 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.366597 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.366612 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.367401 (+ 789us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.367539 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":581,"threads_started":1}
I20260924 11:41:30.378825 8404 raft_consensus.cc:1261] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Refusing update from remote peer 450c3cbc16d94e76857ab513f25d6423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:30.379865 8509 consensus_queue.cc:1035] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:30.383709 8270 raft_consensus.cc:1261] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Refusing update from remote peer 450c3cbc16d94e76857ab513f25d6423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:30.384469 8509 consensus_queue.cc:1035] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:41:30.417367 8337 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.97.2:39793, user_credentials={real_user=slave}} blocked reactor thread for 44214.8us
I20260924 11:41:30.437386 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.430204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51367 (local address 127.0.97.1:35263)
0924 11:41:30.430395 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.430402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.431055 (+ 653us) server_negotiation.cc:407] Connection header received
0924 11:41:30.431332 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.431339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.431427 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.431494 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:30.432316 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.433238 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.435308 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.435995 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.436092 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.436662 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.436763 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.436856 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.436983 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.437053 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:41:30.437521 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.430260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35263 (local address 127.0.97.3:51367)
0924 11:41:30.430957 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:30.430978 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:30.431167 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:30.431667 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:30.431671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:30.431695 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:41:30.432098 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.432110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.433440 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.433444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.435163 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:30.435174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.436312 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.436316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:30.436431 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.436477 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.436856 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.436868 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 11:41:30.436964 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:41:30.437078 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":485,"threads_started":1}
I20260924 11:41:30.475034 7967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:30.475387 7967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:30.478927 7967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Sending DeleteTablet for 3 replicas of tablet 9dcc2904748942e7bafbe0076cb5741c
I20260924 11:41:30.479852 8384 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcc2904748942e7bafbe0076cb5741c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52628
I20260924 11:41:30.480195 8502 tablet_replica.cc:330] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: stopping tablet replica
I20260924 11:41:30.480412 8502 raft_consensus.cc:2229] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:30.480612 8502 raft_consensus.cc:2258] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.480850 8106 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcc2904748942e7bafbe0076cb5741c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43656
I20260924 11:41:30.480871 8245 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcc2904748942e7bafbe0076cb5741c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34808
I20260924 11:41:30.481181 8503 tablet_replica.cc:330] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: stopping tablet replica
I20260924 11:41:30.481266 8501 tablet_replica.cc:330] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: stopping tablet replica
I20260924 11:41:30.481375 8503 raft_consensus.cc:2229] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:30.481427 8501 raft_consensus.cc:2229] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:30.481580 8501 raft_consensus.cc:2258] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.481845 8503 raft_consensus.cc:2258] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.482795 8501 ts_tablet_manager.cc:1905] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.483000 8503 ts_tablet_manager.cc:1905] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.489408 8501 ts_tablet_manager.cc:1918] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:41:30.489547 8501 log.cc:1196] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/9dcc2904748942e7bafbe0076cb5741c
I20260924 11:41:30.489919 8501 ts_tablet_manager.cc:1939] T 9dcc2904748942e7bafbe0076cb5741c P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting consensus metadata
I20260924 11:41:30.490863 7955 catalog_manager.cc:4841] TS d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793): tablet 9dcc2904748942e7bafbe0076cb5741c (table test-table [id=bd4b3398b6e74a0ca6e5597a046d9383]) successfully deleted
I20260924 11:41:30.491710 8502 ts_tablet_manager.cc:1905] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.493206 8503 ts_tablet_manager.cc:1918] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:41:30.493355 8503 log.cc:1196] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/9dcc2904748942e7bafbe0076cb5741c
I20260924 11:41:30.493724 8503 ts_tablet_manager.cc:1939] T 9dcc2904748942e7bafbe0076cb5741c P 450c3cbc16d94e76857ab513f25d6423: Deleting consensus metadata
I20260924 11:41:30.494539 7955 catalog_manager.cc:4841] TS 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263): tablet 9dcc2904748942e7bafbe0076cb5741c (table test-table [id=bd4b3398b6e74a0ca6e5597a046d9383]) successfully deleted
I20260924 11:41:30.498592 8502 ts_tablet_manager.cc:1918] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:41:30.498742 8502 log.cc:1196] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/9dcc2904748942e7bafbe0076cb5741c
I20260924 11:41:30.499171 8502 ts_tablet_manager.cc:1939] T 9dcc2904748942e7bafbe0076cb5741c P 227571db98ce448480b581993c242cc9: Deleting consensus metadata
I20260924 11:41:30.500164 7954 catalog_manager.cc:4841] TS 227571db98ce448480b581993c242cc9 (127.0.97.3:42313): tablet 9dcc2904748942e7bafbe0076cb5741c (table test-table [id=bd4b3398b6e74a0ca6e5597a046d9383]) successfully deleted
Sep 24 11:41:30 dist-test-slave-qhrh krb5kdc[7924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:41:30 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:30 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:41:30.589305 8063 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:30.567316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.0.97.62:40765)
0924 11:41:30.567532 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.567542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.577126 (+ 9584us) server_negotiation.cc:407] Connection header received
0924 11:41:30.577372 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.577378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.577487 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.577564 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:30.579527 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.580658 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.582460 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.583038 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.583124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.589039 (+ 5915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56520: BlockingRecv error: recv got EOF from 127.0.0.1:56520 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 11:41:30 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 11:41:30.609947 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.592340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.97.62:40765)
0924 11:41:30.592503 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.592509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.592972 (+ 463us) server_negotiation.cc:407] Connection header received
0924 11:41:30.593162 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.593167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.593235 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.593329 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:30.594871 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.596006 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.597911 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.598561 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.598698 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.603317 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:30.603324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:30.603329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:30.603364 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:30.606796 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:30.607591 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:30.607597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:30.607601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:30.607765 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:30.608413 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:30.608422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:30.608425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:30.608773 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:30.609010 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.609333 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.609531 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:41:30.613533 7967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:30.614246 7967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:30.621601 8384 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac619ac4914fd3af82409bda0f9983 (DEFAULT_TABLE table=test-table [id=331f91066bad48398d865871ee27ed6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.621719 8245 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac619ac4914fd3af82409bda0f9983 (DEFAULT_TABLE table=test-table [id=331f91066bad48398d865871ee27ed6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.622056 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac619ac4914fd3af82409bda0f9983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.622099 8245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac619ac4914fd3af82409bda0f9983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.624749 8106 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac619ac4914fd3af82409bda0f9983 (DEFAULT_TABLE table=test-table [id=331f91066bad48398d865871ee27ed6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.625180 8106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac619ac4914fd3af82409bda0f9983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.628827 8481 tablet_bootstrap.cc:492] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Bootstrap starting.
I20260924 11:41:30.630474 8481 tablet_bootstrap.cc:654] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.635857 8479 tablet_bootstrap.cc:492] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Bootstrap starting.
I20260924 11:41:30.637794 8479 tablet_bootstrap.cc:654] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.638204 8478 tablet_bootstrap.cc:492] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Bootstrap starting.
I20260924 11:41:30.639698 8478 tablet_bootstrap.cc:654] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.644791 8478 tablet_bootstrap.cc:492] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: No bootstrap required, opened a new log
I20260924 11:41:30.645021 8478 ts_tablet_manager.cc:1397] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260924 11:41:30.646147 8478 raft_consensus.cc:348] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.646570 8478 raft_consensus.cc:374] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.646733 8478 raft_consensus.cc:729] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227571db98ce448480b581993c242cc9, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.647135 8478 consensus_queue.cc:260] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.648191 8478 ts_tablet_manager.cc:1428] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 11:41:30.649550 8481 tablet_bootstrap.cc:492] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: No bootstrap required, opened a new log
I20260924 11:41:30.649708 8481 ts_tablet_manager.cc:1397] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.012s
I20260924 11:41:30.650446 8481 raft_consensus.cc:348] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.650705 8481 raft_consensus.cc:374] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.650804 8481 raft_consensus.cc:729] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a8e15a7dda4a0c8bd0b39696d3a0c8, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.651506 8479 tablet_bootstrap.cc:492] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: No bootstrap required, opened a new log
I20260924 11:41:30.651643 8479 ts_tablet_manager.cc:1397] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260924 11:41:30.652383 8479 raft_consensus.cc:348] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.652603 8479 raft_consensus.cc:374] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.652712 8479 raft_consensus.cc:729] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.652945 8479 consensus_queue.cc:260] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.653482 8479 ts_tablet_manager.cc:1428] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:41:30.653805 8481 consensus_queue.cc:260] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.654304 8481 ts_tablet_manager.cc:1428] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 11:41:30.780714 8509 raft_consensus.cc:482] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:30.780925 8509 raft_consensus.cc:504] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.781898 8509 leader_election.cc:288] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793), 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:30.782160 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac619ac4914fd3af82409bda0f9983" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" is_pre_election: true
I20260924 11:41:30.782414 8270 raft_consensus.cc:2454] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 0.
I20260924 11:41:30.782606 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac619ac4914fd3af82409bda0f9983" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227571db98ce448480b581993c242cc9" is_pre_election: true
I20260924 11:41:30.782869 8404 raft_consensus.cc:2454] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 0.
I20260924 11:41:30.783134 8058 leader_election.cc:302] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450c3cbc16d94e76857ab513f25d6423, d9a8e15a7dda4a0c8bd0b39696d3a0c8; no voters:
I20260924 11:41:30.783694 8509 raft_consensus.cc:2781] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:30.783818 8509 raft_consensus.cc:482] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:30.783903 8509 raft_consensus.cc:3037] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.787014 8509 raft_consensus.cc:504] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.788133 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac619ac4914fd3af82409bda0f9983" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
I20260924 11:41:30.788489 8270 raft_consensus.cc:3037] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.790771 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac619ac4914fd3af82409bda0f9983" candidate_uuid: "450c3cbc16d94e76857ab513f25d6423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227571db98ce448480b581993c242cc9"
I20260924 11:41:30.790931 8404 raft_consensus.cc:3037] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:30.793762 8404 raft_consensus.cc:2454] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 1.
I20260924 11:41:30.794131 8057 leader_election.cc:302] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, 450c3cbc16d94e76857ab513f25d6423; no voters:
I20260924 11:41:30.794425 8535 raft_consensus.cc:2781] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:30.794560 8535 raft_consensus.cc:686] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 1 LEADER]: Becoming Leader. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Running, Role: LEADER
I20260924 11:41:30.794831 8535 consensus_queue.cc:237] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.795154 8270 raft_consensus.cc:2454] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450c3cbc16d94e76857ab513f25d6423 in term 1.
I20260924 11:41:30.800031 8509 leader_election.cc:288] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [CANDIDATE]: Term 1 election: Requested vote from peers d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793), 227571db98ce448480b581993c242cc9 (127.0.97.3:42313)
I20260924 11:41:30.801798 7968 catalog_manager.cc:5495] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "450c3cbc16d94e76857ab513f25d6423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } health_report { overall_health: UNKNOWN } } }
I20260924 11:41:30.841500 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:30.835248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.97.1:35263)
0924 11:41:30.835420 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:30.835425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:30.835444 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:41:30.835571 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:30.835577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:30.835667 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:30.835775 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:30.836332 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.837434 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.839341 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:30.840003 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:30.840140 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:30.840732 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:30.840866 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:30.840997 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:30.841123 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 11:41:30.841209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:41:30.844827 8270 raft_consensus.cc:1261] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Refusing update from remote peer 450c3cbc16d94e76857ab513f25d6423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:30.844885 8404 raft_consensus.cc:1261] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Refusing update from remote peer 450c3cbc16d94e76857ab513f25d6423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:30.845486 8509 consensus_queue.cc:1035] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:30.845757 8535 consensus_queue.cc:1035] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:30.861130 7967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:30.861393 7967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:30.865159 7967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Sending DeleteTablet for 3 replicas of tablet c5ac619ac4914fd3af82409bda0f9983
I20260924 11:41:30.866142 8384 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac619ac4914fd3af82409bda0f9983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52628
I20260924 11:41:30.866413 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac619ac4914fd3af82409bda0f9983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52628
I20260924 11:41:30.866402 8245 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac619ac4914fd3af82409bda0f9983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34808
I20260924 11:41:30.866652 8502 tablet_replica.cc:330] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: stopping tablet replica
I20260924 11:41:30.866716 8501 tablet_replica.cc:330] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: stopping tablet replica
I20260924 11:41:30.866827 8502 raft_consensus.cc:2229] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:30.866922 8501 raft_consensus.cc:2229] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:30.867072 8502 raft_consensus.cc:2258] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.867154 8501 raft_consensus.cc:2258] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.867800 8106 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac619ac4914fd3af82409bda0f9983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43656
I20260924 11:41:30.868064 8503 tablet_replica.cc:330] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: stopping tablet replica
I20260924 11:41:30.868271 8501 ts_tablet_manager.cc:1905] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.868281 8503 raft_consensus.cc:2229] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:30.868628 8503 raft_consensus.cc:2258] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:30.869310 7969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:41:30.869923 7969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:41:30.871603 8503 ts_tablet_manager.cc:1905] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.874219 8501 ts_tablet_manager.cc:1918] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:30.874922 8502 ts_tablet_manager.cc:1905] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:30.877331 8383 tablet_service.cc:1467] Processing CreateTablet for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb (DEFAULT_TABLE table=test-table [id=bb18de945eb04d6fb2e419db21ba7811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.877710 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.878549 8503 ts_tablet_manager.cc:1918] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:30.878705 8503 log.cc:1196] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/c5ac619ac4914fd3af82409bda0f9983
I20260924 11:41:30.879184 8503 ts_tablet_manager.cc:1939] T c5ac619ac4914fd3af82409bda0f9983 P 450c3cbc16d94e76857ab513f25d6423: Deleting consensus metadata
I20260924 11:41:30.880738 7955 catalog_manager.cc:4841] TS 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263): tablet c5ac619ac4914fd3af82409bda0f9983 (table test-table [id=331f91066bad48398d865871ee27ed6d]) successfully deleted
I20260924 11:41:30.881127 8502 ts_tablet_manager.cc:1918] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:41:30.881135 8106 tablet_service.cc:1467] Processing CreateTablet for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb (DEFAULT_TABLE table=test-table [id=bb18de945eb04d6fb2e419db21ba7811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.881246 8502 log.cc:1196] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/c5ac619ac4914fd3af82409bda0f9983
I20260924 11:41:30.881497 8106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.881662 8502 ts_tablet_manager.cc:1939] T c5ac619ac4914fd3af82409bda0f9983 P 227571db98ce448480b581993c242cc9: Deleting consensus metadata
I20260924 11:41:30.882555 7954 catalog_manager.cc:4841] TS 227571db98ce448480b581993c242cc9 (127.0.97.3:42313): tablet c5ac619ac4914fd3af82409bda0f9983 (table test-table [id=331f91066bad48398d865871ee27ed6d]) successfully deleted
W20260924 11:41:30.882956 7954 catalog_manager.cc:4808] TS 227571db98ce448480b581993c242cc9 (127.0.97.3:42313): delete failed for tablet c5ac619ac4914fd3af82409bda0f9983 because the tablet was not found. No further retry: Not found: Tablet not found: c5ac619ac4914fd3af82409bda0f9983
I20260924 11:41:30.884093 8478 tablet_bootstrap.cc:492] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Bootstrap starting.
I20260924 11:41:30.885507 8478 tablet_bootstrap.cc:654] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.887125 8478 tablet_bootstrap.cc:492] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: No bootstrap required, opened a new log
I20260924 11:41:30.887297 8478 ts_tablet_manager.cc:1397] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 11:41:30.887854 8478 raft_consensus.cc:348] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.888056 8478 raft_consensus.cc:374] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.888185 8478 raft_consensus.cc:729] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227571db98ce448480b581993c242cc9, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.888538 8479 tablet_bootstrap.cc:492] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Bootstrap starting.
I20260924 11:41:30.888414 8478 consensus_queue.cc:260] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.876724 8245 tablet_service.cc:1467] Processing CreateTablet for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb (DEFAULT_TABLE table=test-table [id=bb18de945eb04d6fb2e419db21ba7811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:41:30.889400 8245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:30.889490 8478 ts_tablet_manager.cc:1428] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:30.890139 8479 tablet_bootstrap.cc:654] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.891979 8479 tablet_bootstrap.cc:492] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: No bootstrap required, opened a new log
I20260924 11:41:30.892195 8479 ts_tablet_manager.cc:1397] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 11:41:30.892822 8479 raft_consensus.cc:348] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.893049 8479 raft_consensus.cc:374] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.893153 8479 raft_consensus.cc:729] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3cbc16d94e76857ab513f25d6423, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.893371 8479 consensus_queue.cc:260] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.893903 8479 ts_tablet_manager.cc:1428] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:41:30.895289 8481 tablet_bootstrap.cc:492] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Bootstrap starting.
I20260924 11:41:30.896571 8481 tablet_bootstrap.cc:654] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:30.900422 8501 log.cc:1196] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/c5ac619ac4914fd3af82409bda0f9983
I20260924 11:41:30.901180 8501 ts_tablet_manager.cc:1939] T c5ac619ac4914fd3af82409bda0f9983 P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting consensus metadata
I20260924 11:41:30.902199 7955 catalog_manager.cc:4841] TS d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793): tablet c5ac619ac4914fd3af82409bda0f9983 (table test-table [id=331f91066bad48398d865871ee27ed6d]) successfully deleted
I20260924 11:41:30.902947 8481 tablet_bootstrap.cc:492] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: No bootstrap required, opened a new log
I20260924 11:41:30.903098 8481 ts_tablet_manager.cc:1397] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260924 11:41:30.903589 8481 raft_consensus.cc:348] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.903797 8481 raft_consensus.cc:374] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:30.903890 8481 raft_consensus.cc:729] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a8e15a7dda4a0c8bd0b39696d3a0c8, State: Initialized, Role: FOLLOWER
I20260924 11:41:30.904137 8481 consensus_queue.cc:260] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:30.904631 8481 ts_tablet_manager.cc:1428] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 11:41:31.233837 8532 raft_consensus.cc:482] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:41:31.234017 8532 raft_consensus.cc:504] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:31.234627 8532 leader_election.cc:288] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263), d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793)
I20260924 11:41:31.235093 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bc1c0cc6ba49e4bc43aa1de32fa4eb" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3cbc16d94e76857ab513f25d6423" is_pre_election: true
I20260924 11:41:31.235178 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bc1c0cc6ba49e4bc43aa1de32fa4eb" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" is_pre_election: true
I20260924 11:41:31.235301 8131 raft_consensus.cc:2454] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 0.
I20260924 11:41:31.235458 8270 raft_consensus.cc:2454] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 0.
I20260924 11:41:31.235718 8337 leader_election.cc:302] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, 450c3cbc16d94e76857ab513f25d6423; no voters:
I20260924 11:41:31.236004 8532 raft_consensus.cc:2781] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:41:31.236125 8532 raft_consensus.cc:482] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:41:31.236249 8532 raft_consensus.cc:3037] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:31.238096 8532 raft_consensus.cc:504] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:31.238550 8532 leader_election.cc:288] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 election: Requested vote from peers 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263), d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793)
I20260924 11:41:31.239048 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bc1c0cc6ba49e4bc43aa1de32fa4eb" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3cbc16d94e76857ab513f25d6423"
I20260924 11:41:31.239106 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bc1c0cc6ba49e4bc43aa1de32fa4eb" candidate_uuid: "227571db98ce448480b581993c242cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8"
I20260924 11:41:31.239214 8131 raft_consensus.cc:3037] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:31.239318 8270 raft_consensus.cc:3037] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:31.241211 8131 raft_consensus.cc:2454] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 1.
I20260924 11:41:31.241253 8270 raft_consensus.cc:2454] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227571db98ce448480b581993c242cc9 in term 1.
I20260924 11:41:31.241576 8337 leader_election.cc:302] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227571db98ce448480b581993c242cc9, 450c3cbc16d94e76857ab513f25d6423; no voters:
I20260924 11:41:31.241904 8532 raft_consensus.cc:2781] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:31.242261 8532 raft_consensus.cc:686] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 227571db98ce448480b581993c242cc9, State: Running, Role: LEADER
I20260924 11:41:31.242513 8532 consensus_queue.cc:237] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } }
I20260924 11:41:31.244702 7967 catalog_manager.cc:5495] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 227571db98ce448480b581993c242cc9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "227571db98ce448480b581993c242cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227571db98ce448480b581993c242cc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42313 } health_report { overall_health: HEALTHY } } }
I20260924 11:41:31.267913 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:31.263344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.97.3:42313)
0924 11:41:31.263830 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:31.263838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:31.263856 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:41:31.263912 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:31.263917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:31.263989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:31.264100 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:31.264514 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.265277 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.266433 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.266801 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.266908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:31.267249 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:31.267330 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:31.267390 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:31.267504 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:41:31.267572 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":276,"threads_started":1}
I20260924 11:41:31.279099 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:31.274794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53705 (local address 127.0.97.3:42313)
0924 11:41:31.274953 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:31.274961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:31.275312 (+ 351us) server_negotiation.cc:407] Connection header received
0924 11:41:31.275413 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:31.275416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:31.275463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:31.275523 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:41:31.275830 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.276474 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.277724 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.278170 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.278268 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:31.278593 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:41:31.278678 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:41:31.278745 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:31.278853 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 11:41:31.278899 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:41:31.279222 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:31.274750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42313 (local address 127.0.97.3:53705)
0924 11:41:31.275234 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:31.275247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:31.275335 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:31.275545 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:31.275547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:31.275554 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:41:31.275733 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:31.275739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.276567 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.276570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:31.277613 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:31.277624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.278324 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.278329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:31.278439 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:31.278470 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:41:31.278763 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:41:31.278770 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:41:31.278825 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 11:41:31.278909 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":290,"threads_started":1}
I20260924 11:41:31.285364 8270 raft_consensus.cc:1261] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Refusing update from remote peer 227571db98ce448480b581993c242cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:31.285372 8131 raft_consensus.cc:1261] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Refusing update from remote peer 227571db98ce448480b581993c242cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:41:31.285804 8532 consensus_queue.cc:1035] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450c3cbc16d94e76857ab513f25d6423" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:31.286098 8565 consensus_queue.cc:1035] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a8e15a7dda4a0c8bd0b39696d3a0c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:41:31.328420 7969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:31.328639 7969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:41:31.331213 7969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda07ce6a2434a7695e6316874cc0f14: Sending DeleteTablet for 3 replicas of tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb
I20260924 11:41:31.331924 8106 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43656
I20260924 11:41:31.332072 8384 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52628
I20260924 11:41:31.332190 8245 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:41:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34808
I20260924 11:41:31.332346 8502 tablet_replica.cc:330] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: stopping tablet replica
I20260924 11:41:31.332406 8501 tablet_replica.cc:330] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: stopping tablet replica
I20260924 11:41:31.332432 8503 tablet_replica.cc:330] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: stopping tablet replica
I20260924 11:41:31.332532 8502 raft_consensus.cc:2229] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:41:31.332584 8503 raft_consensus.cc:2229] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:31.332580 8501 raft_consensus.cc:2229] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:41:31.332728 8501 raft_consensus.cc:2258] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:31.332738 8503 raft_consensus.cc:2258] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:31.332930 8502 raft_consensus.cc:2258] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:41:31.333993 8502 ts_tablet_manager.cc:1905] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:31.334081 8503 ts_tablet_manager.cc:1905] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:31.340730 8502 ts_tablet_manager.cc:1918] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:41:31.340752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8040
I20260924 11:41:31.340951 8502 log.cc:1196] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/84bc1c0cc6ba49e4bc43aa1de32fa4eb
I20260924 11:41:31.341428 8502 ts_tablet_manager.cc:1939] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 227571db98ce448480b581993c242cc9: Deleting consensus metadata
I20260924 11:41:31.342473 7954 catalog_manager.cc:4841] TS 227571db98ce448480b581993c242cc9 (127.0.97.3:42313): tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb (table test-table [id=bb18de945eb04d6fb2e419db21ba7811]) successfully deleted
I20260924 11:41:31.342653 8503 ts_tablet_manager.cc:1918] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:41:31.342790 8503 log.cc:1196] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/84bc1c0cc6ba49e4bc43aa1de32fa4eb
I20260924 11:41:31.343144 8503 ts_tablet_manager.cc:1939] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P 450c3cbc16d94e76857ab513f25d6423: Deleting consensus metadata
I20260924 11:41:31.343946 7955 catalog_manager.cc:4841] TS 450c3cbc16d94e76857ab513f25d6423 (127.0.97.1:35263): tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb (table test-table [id=bb18de945eb04d6fb2e419db21ba7811]) successfully deleted
I20260924 11:41:31.344420 8501 ts_tablet_manager.cc:1905] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:41:31.348763 8501 ts_tablet_manager.cc:1918] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:41:31.348923 8501 log.cc:1196] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/84bc1c0cc6ba49e4bc43aa1de32fa4eb
I20260924 11:41:31.349263 8501 ts_tablet_manager.cc:1939] T 84bc1c0cc6ba49e4bc43aa1de32fa4eb P d9a8e15a7dda4a0c8bd0b39696d3a0c8: Deleting consensus metadata
I20260924 11:41:31.350148 7955 catalog_manager.cc:4841] TS d9a8e15a7dda4a0c8bd0b39696d3a0c8 (127.0.97.2:39793): tablet 84bc1c0cc6ba49e4bc43aa1de32fa4eb (table test-table [id=bb18de945eb04d6fb2e419db21ba7811]) successfully deleted
Sep 24 11:41:31 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 11:41:31.364037 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:31.341866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.0.97.1:35263)
0924 11:41:31.342049 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:31.342055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:31.350509 (+ 8454us) server_negotiation.cc:407] Connection header received
0924 11:41:31.350650 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:31.350654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:31.350722 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:31.350817 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:31.351884 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.352845 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.353893 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.354497 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.354604 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:31.358220 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:31.358238 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:31.358257 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:31.358298 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:31.361585 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:31.362195 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:31.362204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:31.362208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:31.362334 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:31.362632 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:31.362643 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:31.362646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:31.363063 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:31.363330 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:31.363504 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 11:41:31.363734 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:41:31.364756 8171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:31.480952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8040
I20260924 11:41:31.502859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8180
Sep 24 11:41:31 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 11:41:31.517781 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:31.503584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.0.97.2:39793)
0924 11:41:31.504049 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:31.504053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:31.504067 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:41:31.504114 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:31.504117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:31.504241 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:31.504350 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:31.505521 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.506208 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.507533 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.508073 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.508290 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:31.512270 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:31.512284 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:31.512308 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:31.512340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:31.515079 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:31.515665 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:31.515673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:31.515678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:31.515802 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:31.516136 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:31.516171 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:31.516177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:31.516562 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:31.516851 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:31.517216 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 11:41:31.517417 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260924 11:41:31.518554 8310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:31.638357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8180
I20260924 11:41:31.666303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8319
Sep 24 11:41:31 dist-test-slave-qhrh krb5kdc[7924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 11:41:31.685026 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:31.667159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.97.3:42313)
0924 11:41:31.667391 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:31.667398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:31.667715 (+ 317us) server_negotiation.cc:407] Connection header received
0924 11:41:31.667880 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:31.667886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:31.667957 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:31.668067 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:31.669779 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.671038 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.672886 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:31.673518 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:31.673631 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:31.678824 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:31.678844 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:31.678870 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:31.678912 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:31.682493 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:31.683164 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:31.683173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:31.683177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:31.683316 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:31.683608 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:31.683612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:31.683616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:31.684020 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:31.684317 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:31.684506 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:41:31.684685 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:41:31.685739 8449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:31.828408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8319
I20260924 11:41:31.853382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 7937
I20260924 11:41:31.854454 8009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:31.962759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 7937
2026-09-24T11:41:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5519 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:32 dist-test-slave-qhrh krb5kdc[8583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:32 dist-test-slave-qhrh krb5kdc[8583](info): set up 2 sockets
Sep 24 11:41:32 dist-test-slave-qhrh krb5kdc[8583](info): commencing operation
krb5kdc: starting...
W20260924 11:41:33.014134 388 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:33 dist-test-slave-qhrh krb5kdc[8583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:41:33.212879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39139
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:33.524565 8596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:33.524907 8596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:33.525069 8596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:33.532543 8596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:33.532672 8596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:33.532783 8596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:33.532886 8596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:33.532970 8596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:33.542483 8596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:33.543665 8596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:33.545003 8596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:33.556572 8604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:33.556643 8602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:33.556496 8601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:33.557299 8596 server_base.cc:1034] running on GCE node
I20260924 11:41:33.557960 8596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:33.559203 8596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:33.560426 8596 hybrid_clock.cc:648] HybridClock initialized: now 1790250093560393 us; error 26 us; skew 500 ppm
I20260924 11:41:33.560793 8596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T11:41:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1675 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:33 dist-test-slave-qhrh krb5kdc[8610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:33 dist-test-slave-qhrh krb5kdc[8610](info): set up 2 sockets
Sep 24 11:41:33 dist-test-slave-qhrh krb5kdc[8610](info): commencing operation
krb5kdc: starting...
W20260924 11:41:34.692718 388 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:34 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:41:34.875394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36591
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:35.171387 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:35.171638 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:35.171717 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:35.178196 8623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:35.178287 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:35.178330 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:35.178377 8623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:35.178421 8623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:35.187206 8623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:35.188238 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:35.189352 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:35.200559 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:35.200333 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:35.200512 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:35.201462 8623 server_base.cc:1034] running on GCE node
I20260924 11:41:35.202172 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:35.203331 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:35.204559 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790250095204530 us; error 49 us; skew 500 ppm
I20260924 11:41:35.204855 8623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:35 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:35.207618 8623 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:41:35.208927 8623 webserver.cc:460] Webserver started at http://127.0.97.62:34205/ using document root <none> and password file <none>
I20260924 11:41:35.209545 8623 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:35.209683 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:35.209975 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:35.212669 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "186d846019f30cd7e81f2910ce8b1c3b"
server_key_iv: "6413becf6cee26e61803c959b2d72ea6"
server_key_version: "encryptionkey@0"
I20260924 11:41:35.213375 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "186d846019f30cd7e81f2910ce8b1c3b"
server_key_iv: "6413becf6cee26e61803c959b2d72ea6"
server_key_version: "encryptionkey@0"
I20260924 11:41:35.217777 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:41:35.220790 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:35.221901 8623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:35.222095 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "186d846019f30cd7e81f2910ce8b1c3b"
server_key_iv: "6413becf6cee26e61803c959b2d72ea6"
server_key_version: "encryptionkey@0"
I20260924 11:41:35.222287 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:35.236930 8623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:35.240389 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:35.240630 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:35.257164 8623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36591
I20260924 11:41:35.257195 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36591 every 8 connection(s)
I20260924 11:41:35.258378 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:35.260675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8623
I20260924 11:41:35.260913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:35.261729 388 external_mini_cluster.cc:1428] Setting key 3247ae4a33d926fdc235033ae4a13611
I20260924 11:41:35.262444 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:35.273627 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Bootstrap starting.
Sep 24 11:41:35 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:35.276796 8690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:35.277882 8690 log.cc:824] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:35.280463 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: No bootstrap required, opened a new log
I20260924 11:41:35.282399 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:35.263617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.97.62:36591)
0924 11:41:35.264358 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:35.264368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:35.264403 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:41:35.266569 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:35.266588 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:35.266941 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:35.267320 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:35.268779 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:35.269947 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:35.271121 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:35.271785 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:35.271962 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:35.275772 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:35.275779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:35.275791 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:35.275821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:35.278786 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:35.279433 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:35.279444 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:35.279452 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:35.279611 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:35.279957 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:35.279963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:35.279966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:35.280458 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:35.280742 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:35.281146 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 11:41:35.281460 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":316,"threads_started":1}
I20260924 11:41:35.287533 8690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:35.287863 8690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:35.287989 8690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2d89ba3a8c46b2a292f30ff4ef49aa, State: Initialized, Role: FOLLOWER
I20260924 11:41:35.288582 8690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:35.288775 8690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:35.288857 8690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:35.288968 8690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:35.291433 8690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:35.291826 8690 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2d89ba3a8c46b2a292f30ff4ef49aa; no voters:
I20260924 11:41:35.292354 8690 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:35.292476 8695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:35.293138 8695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 LEADER]: Becoming Leader. State: Replica: 9c2d89ba3a8c46b2a292f30ff4ef49aa, State: Running, Role: LEADER
I20260924 11:41:35.293637 8695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:35.294454 8690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:35.296326 8696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } } }
I20260924 11:41:35.296772 8696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:35.296905 8697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2d89ba3a8c46b2a292f30ff4ef49aa. Latest consensus state: current_term: 1 leader_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } } }
I20260924 11:41:35.297187 8697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:35.300958 8703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:35.306107 8703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:35.314900 8703 catalog_manager.cc:1347] Generated new cluster ID: 5b166cd669fa40b5bd4bbc748e8a52f8
I20260924 11:41:35.315024 8703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:35.323045 8703 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:35.324558 8703 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:35.330729 8703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Generated new TSK 0
I20260924 11:41:35.331436 8703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:41:35.383219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36591
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:35.706301 8716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:35.706534 8716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:35.706718 8716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:35.713466 8716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:35.713548 8716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:35.713683 8716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:35.722605 8716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:35.723668 8716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:35.725029 8716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:35.737128 8721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:35.737262 8724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:35.737246 8722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:35.737838 8716 server_base.cc:1034] running on GCE node
I20260924 11:41:35.738373 8716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:35.739185 8716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:35.740391 8716 hybrid_clock.cc:648] HybridClock initialized: now 1790250095740355 us; error 48 us; skew 500 ppm
I20260924 11:41:35.740664 8716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:35 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:35.743361 8716 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:41:35.744570 8716 webserver.cc:460] Webserver started at http://127.0.97.1:42737/ using document root <none> and password file <none>
I20260924 11:41:35.745175 8716 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:35.745313 8716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:35.745620 8716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:35.748248 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "237a9628150048648280e576f391c8d6"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "56908f84cc93a92e0b95c99e3bc7f4a0"
server_key_iv: "31e61289b91d92d4d6bcf87a8008eb9f"
server_key_version: "encryptionkey@0"
I20260924 11:41:35.748974 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "237a9628150048648280e576f391c8d6"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "56908f84cc93a92e0b95c99e3bc7f4a0"
server_key_iv: "31e61289b91d92d4d6bcf87a8008eb9f"
server_key_version: "encryptionkey@0"
I20260924 11:41:35.753167 8716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:41:35.756007 8731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:35.757190 8716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:35.757349 8716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "237a9628150048648280e576f391c8d6"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "56908f84cc93a92e0b95c99e3bc7f4a0"
server_key_iv: "31e61289b91d92d4d6bcf87a8008eb9f"
server_key_version: "encryptionkey@0"
I20260924 11:41:35.757534 8716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:35.774302 8716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:35.777808 8716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:35.778076 8716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:35.778867 8716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:35.780062 8716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:35.780229 8716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:35.780366 8716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:35.780436 8716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:35.810525 8716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46287
I20260924 11:41:35.810566 8843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46287 every 8 connection(s)
I20260924 11:41:35.811596 8716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:35.813032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8716
I20260924 11:41:35.813241 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:35.813972 388 external_mini_cluster.cc:1428] Setting key 7cbaa5aee6b9830421bfe3b411edde8a
Sep 24 11:41:35 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:35.832393 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:35.814086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38029 (local address 127.0.97.62:36591)
0924 11:41:35.814529 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:35.814535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:35.815521 (+ 986us) server_negotiation.cc:407] Connection header received
0924 11:41:35.817204 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:35.817212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:35.817304 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:35.817410 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:35.819448 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:35.820130 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:35.822599 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:35.822990 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:35.823081 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:35.826897 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:35.826902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:35.826905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:35.826942 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:35.829388 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:35.830146 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:35.830153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:35.830159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:35.830366 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:35.830748 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:35.830754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:35.830756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:35.831087 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:35.831233 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:35.831812 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 11:41:35.832007 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":234,"threads_started":1}
I20260924 11:41:35.833307 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:35.814595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:38029)
0924 11:41:35.815370 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:35.815415 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:35.816967 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:35.817798 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:35.817812 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:35.818563 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:35.819293 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:35.819316 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:35.820341 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:35.820349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:35.822459 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:35.822477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:35.823198 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:35.823205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:35.823496 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:35.824169 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:35.824206 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:35.826693 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:35.829516 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:35.829524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:35.829537 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:35.830036 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:35.830462 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:35.830468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:35.830471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:35.830643 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:35.831214 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:35.831221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:35.831507 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 11:41:35.831746 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 11:41:35.832210 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:41:35.834759 8844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36591
I20260924 11:41:35.835006 8844 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:35.835691 8844 heartbeater.cc:511] Master 127.0.97.62:36591 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 11:41:35.837913 8655 ts_manager.cc:194] Registered new tserver with Master: 237a9628150048648280e576f391c8d6 (127.0.97.1:46287)
I20260924 11:41:35.839994 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:41:35.859314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36591
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:36.224344 8850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:36.224587 8850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:36.224730 8850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:36.231460 8850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:36.231549 8850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:36.231701 8850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:36.240578 8850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:36.241670 8850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:36.242885 8850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:36.258057 8855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:36.258152 8858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:36.259027 8850 server_base.cc:1034] running on GCE node
W20260924 11:41:36.260627 8856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:36.261384 8850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:36.262367 8850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:36.263602 8850 hybrid_clock.cc:648] HybridClock initialized: now 1790250096263543 us; error 52 us; skew 500 ppm
I20260924 11:41:36.263943 8850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:36 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:36.267218 8850 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:41:36.275619 8850 webserver.cc:460] Webserver started at http://127.0.97.2:39865/ using document root <none> and password file <none>
I20260924 11:41:36.276345 8850 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:36.276500 8850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:36.276798 8850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:36.279677 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "8eb6b587696b4c59b5f423e4205d1f07"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "d071c7d7b4690c2db03a6c0dfbb0f2ec"
server_key_iv: "cbad1b90de0970d802d25fedfcec32c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:36.280498 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8eb6b587696b4c59b5f423e4205d1f07"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "d071c7d7b4690c2db03a6c0dfbb0f2ec"
server_key_iv: "cbad1b90de0970d802d25fedfcec32c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:36.285624 8850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 11:41:36.289083 8865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:36.290365 8850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:41:36.290526 8850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "8eb6b587696b4c59b5f423e4205d1f07"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "d071c7d7b4690c2db03a6c0dfbb0f2ec"
server_key_iv: "cbad1b90de0970d802d25fedfcec32c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:36.290732 8850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:36.311030 8850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:36.314967 8850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:36.315228 8850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:36.316125 8850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:36.317394 8850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:36.317493 8850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:36.317627 8850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:36.317696 8850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:36.348886 8850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46435
I20260924 11:41:36.348959 8977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46435 every 8 connection(s)
I20260924 11:41:36.349994 8850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:36.355898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8850
I20260924 11:41:36.356086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:36.356627 388 external_mini_cluster.cc:1428] Setting key fa5bedfd9e4326079a104627d19ad8c6
Sep 24 11:41:36 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:36.369443 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:36.351985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49429 (local address 127.0.97.62:36591)
0924 11:41:36.352388 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:36.352393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:36.353195 (+ 802us) server_negotiation.cc:407] Connection header received
0924 11:41:36.354729 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:36.354732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:36.354789 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:36.354879 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:36.356508 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.357278 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.359121 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.359511 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.359591 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:36.363279 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:36.363286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:36.363293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:36.363331 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:36.366214 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:36.366988 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:36.366996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:36.367002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:36.367178 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:36.367577 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:36.367585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:36.367591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:36.367952 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:36.368102 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:36.368743 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 11:41:36.368975 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":211,"threads_started":1}
I20260924 11:41:36.369910 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:36.352490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:49429)
0924 11:41:36.353059 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:36.353096 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:36.354519 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:36.355052 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:36.355058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:36.355509 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:36.356347 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:36.356372 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.357421 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.357425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:36.358970 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:36.358984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.359720 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.359729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:36.360044 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:36.360721 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:36.360751 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:36.363065 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:36.366344 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:36.366352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:36.366369 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:36.366851 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:36.367290 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:36.367294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:36.367298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:36.367465 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:36.368062 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:36.368067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:36.368385 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 11:41:36.368584 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 11:41:36.368906 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":226,"threads_started":1}
I20260924 11:41:36.371184 8978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36591
I20260924 11:41:36.371459 8978 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:36.372254 8978 heartbeater.cc:511] Master 127.0.97.62:36591 requested a full tablet report, sending...
I20260924 11:41:36.373850 8655 ts_manager.cc:194] Registered new tserver with Master: 8eb6b587696b4c59b5f423e4205d1f07 (127.0.97.2:46435)
I20260924 11:41:36.374629 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49429
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:41:36.403380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36591
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:36.700812 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:36.701081 8984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:36.701186 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:36.707711 8984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:36.707803 8984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:36.707980 8984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:36.716653 8984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:36.717625 8984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:36.718824 8984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:36.731385 8992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:36.731455 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:36.731508 8989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:36.732093 8984 server_base.cc:1034] running on GCE node
I20260924 11:41:36.732650 8984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:36.733599 8984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:36.734820 8984 hybrid_clock.cc:648] HybridClock initialized: now 1790250096734778 us; error 52 us; skew 500 ppm
I20260924 11:41:36.735102 8984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:36 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:36.737875 8984 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:41:36.739157 8984 webserver.cc:460] Webserver started at http://127.0.97.3:39681/ using document root <none> and password file <none>
I20260924 11:41:36.739774 8984 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:36.739908 8984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:36.740252 8984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:36.742653 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "83727ed611a74960935899729a6ff3e8"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "69f8fc26fa66485757ad8db1045ffc20"
server_key_iv: "9db500284dade8f161b6a250c719bbb6"
server_key_version: "encryptionkey@0"
I20260924 11:41:36.743328 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "83727ed611a74960935899729a6ff3e8"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "69f8fc26fa66485757ad8db1045ffc20"
server_key_iv: "9db500284dade8f161b6a250c719bbb6"
server_key_version: "encryptionkey@0"
I20260924 11:41:36.747699 8984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:41:36.750849 8999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:36.751916 8984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:36.752063 8984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "83727ed611a74960935899729a6ff3e8"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "69f8fc26fa66485757ad8db1045ffc20"
server_key_iv: "9db500284dade8f161b6a250c719bbb6"
server_key_version: "encryptionkey@0"
I20260924 11:41:36.752298 8984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:36.772718 8984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:36.775961 8984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:36.776238 8984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:36.777024 8984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:36.778162 8984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:36.778272 8984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:36.778409 8984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:36.778479 8984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:36.807437 8984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36283
I20260924 11:41:36.807518 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36283 every 8 connection(s)
I20260924 11:41:36.808636 8984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:36.809561 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 8984
I20260924 11:41:36.809753 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:36.810199 388 external_mini_cluster.cc:1428] Setting key 43d2d60cd04c627d7d87a79b2e75d60a
Sep 24 11:41:36 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:41:36.826959 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:36.812320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47889 (local address 127.0.97.62:36591)
0924 11:41:36.812502 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:36.812509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:36.813029 (+ 520us) server_negotiation.cc:407] Connection header received
0924 11:41:36.814846 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:36.814851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:36.814936 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:36.815036 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:36.816274 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.816946 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.818305 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.818685 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.818774 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:36.821852 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:36.821857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:36.821860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:36.821887 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:36.824065 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:36.824848 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:36.824855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:36.824859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:36.825012 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:36.825495 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:36.825500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:36.825503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:36.825824 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:36.825929 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:36.826561 (+ 632us) server_negotiation.cc:299] Negotiation successful
0924 11:41:36.826749 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:41:36.827704 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:36.811937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:47889)
0924 11:41:36.812876 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:36.812935 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:36.814599 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:36.815175 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:36.815182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:36.815610 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:36.816098 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:36.816112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.817060 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.817063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:36.818191 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:36.818201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.818854 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.818858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:36.819031 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:36.819672 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:36.819691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:36.821666 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:36.824271 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:36.824277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:36.824286 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:36.824716 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:36.825145 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:41:36.825150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:41:36.825154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:41:36.825389 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:41:36.825926 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:36.825931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:36.826189 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 11:41:36.826442 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 11:41:36.826751 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":445,"threads_started":1}
I20260924 11:41:36.828841 9112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36591
I20260924 11:41:36.829094 9112 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:36.829687 9112 heartbeater.cc:511] Master 127.0.97.62:36591 requested a full tablet report, sending...
I20260924 11:41:36.830850 8655 ts_manager.cc:194] Registered new tserver with Master: 83727ed611a74960935899729a6ff3e8 (127.0.97.3:36283)
I20260924 11:41:36.831600 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47889
I20260924 11:41:36.837472 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:41:36.843087 8844 heartbeater.cc:503] Master 127.0.97.62:36591 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 11:41:36.950255 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8716
Sep 24 11:41:36 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:41:36.965037 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:36.951127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.0.97.1:46287)
0924 11:41:36.951838 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:36.951844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:36.951869 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:41:36.951942 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:36.951948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:36.952225 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:36.952364 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:36.953514 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.954456 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.955714 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:36.956371 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:36.956478 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:36.960132 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:36.960177 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:36.960196 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:36.960225 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:36.962885 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:36.963458 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:36.963464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:36.963466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:36.963573 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:36.963830 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:36.963832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:36.963834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:36.964128 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:36.964379 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:36.964595 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 11:41:36.964773 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":418,"threads_started":1}
I20260924 11:41:36.966233 8839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:37.048131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8716
I20260924 11:41:37.064049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8850
Sep 24 11:41:37 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:41:37.078840 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:37.064923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.0.97.2:46435)
0924 11:41:37.065349 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:37.065355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:37.065376 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:37.065432 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:37.065439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:37.065624 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:37.065790 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:37.066894 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:37.067840 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:37.069203 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:37.069743 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:37.069876 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:37.073567 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:37.073576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:37.073589 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:37.073616 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:37.076271 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:37.076869 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:37.076878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:37.076881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:37.077020 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:37.077364 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:37.077370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:37.077373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:37.077673 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:37.077970 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:37.078263 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 11:41:37.078509 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":217,"threads_started":1}
I20260924 11:41:37.080014 8973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:37.159377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8850
I20260924 11:41:37.174775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8984
Sep 24 11:41:37 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:41:37.189436 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:37.175702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.0.97.3:36283)
0924 11:41:37.176315 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:37.176322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:37.176350 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:41:37.176407 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:37.176413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:37.176634 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:37.176760 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:37.177980 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:37.178937 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:37.180188 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:37.180737 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:37.180827 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:37.184537 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:37.184546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:37.184559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:37.184583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:37.187159 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:37.187681 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:37.187687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:37.187689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:37.187797 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:37.188084 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:37.188087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:37.188089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:37.188455 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:37.188670 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:37.188861 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 11:41:37.189032 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":268,"threads_started":1}
I20260924 11:41:37.190635 9107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:37.273303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8984
I20260924 11:41:37.288376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 8623
I20260924 11:41:37.289515 8685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:37.385159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 8623
I20260924 11:41:37.404829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36591
--webserver_interface=127.0.97.62
--webserver_port=34205
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:41:37.703938 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:37.704319 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:37.704473 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:37.711766 9129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:37.711875 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:37.711952 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:37.712033 9129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:37.712092 9129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:37.721030 9129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:37.722254 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:37.723448 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:37.734549 9134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:37.734517 9137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:37.734486 9135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:37.735319 9129 server_base.cc:1034] running on GCE node
I20260924 11:41:37.735949 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:37.737252 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:37.738471 9129 hybrid_clock.cc:648] HybridClock initialized: now 1790250097738420 us; error 42 us; skew 500 ppm
I20260924 11:41:37.738771 9129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:37 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:37.741638 9129 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:41:37.742930 9129 webserver.cc:460] Webserver started at http://127.0.97.62:34205/ using document root <none> and password file <none>
I20260924 11:41:37.743522 9129 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:37.743675 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:37.748406 9129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:41:37.750955 9144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:37.752229 9129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:41:37.752381 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "186d846019f30cd7e81f2910ce8b1c3b"
server_key_iv: "6413becf6cee26e61803c959b2d72ea6"
server_key_version: "encryptionkey@0"
I20260924 11:41:37.753144 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:37.773475 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:37.776772 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:37.777019 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:37.793226 9129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36591
I20260924 11:41:37.793259 9195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36591 every 8 connection(s)
I20260924 11:41:37.794520 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:37.800573 9196 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:41:37.802332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9129
I20260924 11:41:37.803187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46287
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36591
--webserver_port=42737
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:41:37.804987 9196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Bootstrap starting.
I20260924 11:41:37.823971 9196 log.cc:824] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:37.831825 9196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:41:37.832340 9196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Bootstrap complete.
I20260924 11:41:37.841054 9196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:37.841804 9196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2d89ba3a8c46b2a292f30ff4ef49aa, State: Initialized, Role: FOLLOWER
I20260924 11:41:37.842386 9196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:37.842612 9196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:37.842774 9196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:37.842957 9196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:41:37.846354 9196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:37.846793 9196 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2d89ba3a8c46b2a292f30ff4ef49aa; no voters:
I20260924 11:41:37.847414 9196 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:41:37.847534 9200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:41:37.848699 9200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [term 2 LEADER]: Becoming Leader. State: Replica: 9c2d89ba3a8c46b2a292f30ff4ef49aa, State: Running, Role: LEADER
I20260924 11:41:37.849238 9200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } }
I20260924 11:41:37.849468 9196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:37.851788 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2d89ba3a8c46b2a292f30ff4ef49aa. Latest consensus state: current_term: 2 leader_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } } }
I20260924 11:41:37.852102 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:37.852533 9201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d89ba3a8c46b2a292f30ff4ef49aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36591 } } }
I20260924 11:41:37.852823 9201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:37.863556 9207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:37.868914 9207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:37.870546 9207 catalog_manager.cc:1259] Loaded cluster ID: 5b166cd669fa40b5bd4bbc748e8a52f8
I20260924 11:41:37.870733 9207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:37.875334 9207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:37.877048 9207 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c2d89ba3a8c46b2a292f30ff4ef49aa: Loaded TSK: 0
I20260924 11:41:37.878875 9207 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:41:38.151779 9198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:38.152050 9198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:38.152220 9198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:38.159236 9198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:38.159384 9198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:38.159564 9198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:38.168712 9198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42737
--tserver_master_addrs=127.0.97.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:38.169903 9198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:38.171154 9198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:38.184744 9223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:38.184741 9222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:38.185352 9225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:38.185745 9198 server_base.cc:1034] running on GCE node
I20260924 11:41:38.186355 9198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:38.187330 9198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:38.188529 9198 hybrid_clock.cc:648] HybridClock initialized: now 1790250098188475 us; error 47 us; skew 500 ppm
I20260924 11:41:38.188818 9198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250098, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:38.191883 9198 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 11:41:38.193454 9198 webserver.cc:460] Webserver started at http://127.0.97.1:42737/ using document root <none> and password file <none>
I20260924 11:41:38.194113 9198 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:38.194259 9198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:38.199455 9198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 11:41:38.202083 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:38.203291 9198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 11:41:38.203454 9198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "237a9628150048648280e576f391c8d6"
format_stamp: "Formatted at 2026-09-24 11:41:35 on dist-test-slave-qhrh"
server_key: "56908f84cc93a92e0b95c99e3bc7f4a0"
server_key_iv: "31e61289b91d92d4d6bcf87a8008eb9f"
server_key_version: "encryptionkey@0"
I20260924 11:41:38.204387 9198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:38.225934 9198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:38.229890 9198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:38.230136 9198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:38.231035 9198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:38.232437 9198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:38.232563 9198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:38.232692 9198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:38.232779 9198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:38.263804 9198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46287
I20260924 11:41:38.263891 9344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46287 every 8 connection(s)
I20260924 11:41:38.265169 9198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:38.274652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9198
I20260924 11:41:38.275336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46435
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36591
--webserver_port=39865
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:38.283279 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:38.267980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:55091)
0924 11:41:38.268775 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:38.268823 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:38.270631 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:38.271772 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:38.271778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:38.272248 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:38.272726 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.272742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.274070 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.274073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.276101 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.276119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.277090 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.277100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.277410 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.278028 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:38.278060 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:38.282522 (+ 4462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":373,"threads_started":1}
W20260924 11:41:38.283942 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:38.267854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55091 (local address 127.0.97.62:36591)
0924 11:41:38.268510 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:38.268521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:38.268932 (+ 411us) server_negotiation.cc:407] Connection header received
0924 11:41:38.271065 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:38.271077 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:38.271373 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:38.271656 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:38.272873 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.273947 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.276252 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.276862 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.276985 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.282904 (+ 5919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55091: BlockingRecv error: recv got EOF from 127.0.97.1:55091 (error 108)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
W20260924 11:41:38.284044 9345 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36591 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36591: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:38.292464 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:38.284765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:37595)
0924 11:41:38.284913 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:38.284930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:38.285004 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:38.285272 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:38.285275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:38.285536 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:38.285653 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.285660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.287074 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.287077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.287916 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.287925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.288916 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.288920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.289032 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.289579 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:38.289593 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:38.292274 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:41:38.292768 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:38.284840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37595 (local address 127.0.97.62:36591)
0924 11:41:38.284995 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:38.285002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:38.285020 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:41:38.285067 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:38.285074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:38.285141 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:38.285252 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:38.285778 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.286913 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.288046 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.288791 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.288927 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.292523 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37595: BlockingRecv error: recv got EOF from 127.0.97.1:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:38.300305 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:38.293244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:55861)
0924 11:41:38.293400 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:38.293414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:38.293480 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:38.293861 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:38.293866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:38.294139 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:38.294258 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.294265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.295184 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.295191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.296066 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.296076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.296832 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.296848 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.297006 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.297553 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:38.297568 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:38.300049 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:41:38.300549 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:38.293456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55861 (local address 127.0.97.62:36591)
0924 11:41:38.293622 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:38.293627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:38.293648 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:38.293681 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:38.293684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:38.293740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:38.293844 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:38.294387 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.295050 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.296264 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.296689 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.296831 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.300309 (+ 3478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55861: BlockingRecv error: recv got EOF from 127.0.97.1:55861 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:41:38.300648 9345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:41:38.582365 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:38.582587 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:38.582665 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:38.589520 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:38.589604 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:38.589700 9349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:38.598433 9349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39865
--tserver_master_addrs=127.0.97.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:38.599476 9349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:38.600665 9349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:38.612901 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:38.612936 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:38.612932 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:38.613648 9349 server_base.cc:1034] running on GCE node
I20260924 11:41:38.614131 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:38.614907 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:38.616110 9349 hybrid_clock.cc:648] HybridClock initialized: now 1790250098616075 us; error 26 us; skew 500 ppm
I20260924 11:41:38.616467 9349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250098, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:38.627964 9349 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 11:41:38.629343 9349 webserver.cc:460] Webserver started at http://127.0.97.2:39865/ using document root <none> and password file <none>
I20260924 11:41:38.629954 9349 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:38.630113 9349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:38.634894 9349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:41:38.637382 9364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:38.638571 9349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:38.638738 9349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "8eb6b587696b4c59b5f423e4205d1f07"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "d071c7d7b4690c2db03a6c0dfbb0f2ec"
server_key_iv: "cbad1b90de0970d802d25fedfcec32c2"
server_key_version: "encryptionkey@0"
I20260924 11:41:38.639595 9349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:38.659121 9349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:38.662490 9349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:38.662727 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:38.663532 9349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:38.664656 9349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:38.664764 9349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:38.664892 9349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:38.664968 9349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:38.695746 9349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46435
I20260924 11:41:38.695827 9476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46435 every 8 connection(s)
I20260924 11:41:38.697072 9349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:38.707541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9349
I20260924 11:41:38.708086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36283
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36591
--webserver_port=39681
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:41:38.711529 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:38.699273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45679 (local address 127.0.97.62:36591)
0924 11:41:38.699440 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:38.699456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:38.700420 (+ 964us) server_negotiation.cc:407] Connection header received
0924 11:41:38.702213 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:38.702218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:38.702305 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:38.702410 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:38.703686 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.704411 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.705838 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.706228 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.706356 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.711343 (+ 4987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45679: BlockingRecv error: recv got EOF from 127.0.97.2:45679 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:41:38.712379 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:38.699614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:45679)
0924 11:41:38.700262 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:38.700314 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:38.701957 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:38.702628 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:38.702636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:38.703056 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:38.703512 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.703526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.704563 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.704567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.705686 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.705697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.706438 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.706446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.706648 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.707242 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:38.707266 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:38.711218 (+ 3952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":239,"threads_started":1}
W20260924 11:41:38.712968 9477 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36591 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36591: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:41:38.720991 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:38.713555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48291 (local address 127.0.97.62:36591)
0924 11:41:38.713721 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:38.713725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:38.713740 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:41:38.713846 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:38.713848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:38.713922 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:38.713990 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:38.714737 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.715445 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.716522 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.717255 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.717383 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.720842 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48291: BlockingRecv error: recv got EOF from 127.0.97.2:48291 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:41:38.721004 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:38.713427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:48291)
0924 11:41:38.713581 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:38.713599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:38.713721 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:38.714038 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:38.714044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:38.714433 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:38.714614 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.714626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.715569 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.715572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.716389 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.716398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.717378 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.717382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.717490 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.717979 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:38.717993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:38.720671 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:38 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:41:38.730749 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:38.721795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38533 (local address 127.0.97.62:36591)
0924 11:41:38.721908 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:38.721911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:38.722053 (+ 142us) server_negotiation.cc:407] Connection header received
0924 11:41:38.722172 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:38.722175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:38.722221 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:38.722301 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:38.723041 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.723764 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.725763 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.726139 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.726255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.730619 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38533: BlockingRecv error: recv got EOF from 127.0.97.2:38533 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 11:41:38.730813 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:38.721702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:38533)
0924 11:41:38.721974 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:38.721992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:38.722088 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:38.722354 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:38.722360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:38.722729 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:38.722918 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.722931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.723909 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.723916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.725594 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:38.725610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:38.726297 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:38.726306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:38.726517 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:38.727179 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:38.727204 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:38.730492 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154}
W20260924 11:41:38.731053 9477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:41:39.018559 9480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:39.018913 9480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:39.019065 9480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:39.027487 9480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:39.027613 9480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:39.027781 9480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:39.036736 9480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39681
--tserver_master_addrs=127.0.97.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:39.037851 9480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:39.039057 9480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:39.051007 9485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:39.051069 9488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:39.051012 9486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:39.051553 9480 server_base.cc:1034] running on GCE node
I20260924 11:41:39.052073 9480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:39.052917 9480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:39.054116 9480 hybrid_clock.cc:648] HybridClock initialized: now 1790250099054078 us; error 52 us; skew 500 ppm
I20260924 11:41:39.054375 9480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250099, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:41:39.057435 9480 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 11:41:39.058693 9480 webserver.cc:460] Webserver started at http://127.0.97.3:39681/ using document root <none> and password file <none>
I20260924 11:41:39.059360 9480 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:39.059495 9480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:39.064456 9480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:41:39.066834 9495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:39.068002 9480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:39.068188 9480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "83727ed611a74960935899729a6ff3e8"
format_stamp: "Formatted at 2026-09-24 11:41:36 on dist-test-slave-qhrh"
server_key: "69f8fc26fa66485757ad8db1045ffc20"
server_key_iv: "9db500284dade8f161b6a250c719bbb6"
server_key_version: "encryptionkey@0"
I20260924 11:41:39.069058 9480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:39.106949 9480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:39.111229 9480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:39.111528 9480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:39.112357 9480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:39.113575 9480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:39.113694 9480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:39.113806 9480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:39.113888 9480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:39.144626 9480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36283
I20260924 11:41:39.144713 9607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36283 every 8 connection(s)
I20260924 11:41:39.145918 9480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:39.150426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9480
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:41:39.161216 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:39.148215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54339 (local address 127.0.97.62:36591)
0924 11:41:39.148420 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:39.148426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:39.149618 (+ 1192us) server_negotiation.cc:407] Connection header received
0924 11:41:39.151492 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:39.151497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:39.151572 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:39.151638 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:39.153311 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.153994 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.155301 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.155829 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.156024 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.160996 (+ 4972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54339: BlockingRecv error: recv got EOF from 127.0.97.3:54339 (error 108)
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":39680}
I20260924 11:41:39.162070 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:39.148738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:54339)
0924 11:41:39.149465 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:39.149499 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:39.151163 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:39.152122 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:39.152133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:39.152677 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:39.153167 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.153182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.154129 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.154133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.155180 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.155190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.156073 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.156082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.156367 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.157087 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:39.157106 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:39.160861 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
W20260924 11:41:39.162753 9608 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36591 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36591: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260924 11:41:39.165051 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:39.151246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.97.62:36591)
0924 11:41:39.151926 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:39.151933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:39.151949 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:41:39.151988 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:39.151992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:39.152050 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:39.152169 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:39.153447 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.154143 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.155281 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.155775 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.155946 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.158560 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:39.158567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:39.158580 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:39.158614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:39.162036 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:39.162727 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:39.162733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:39.162738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:39.162924 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:41:39.163252 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:41:39.163257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:41:39.163260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:41:39.163772 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:39.164056 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:39.164388 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 11:41:39.164744 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":581,"thread_start_us":483,"threads_started":1}
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:39.170749 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:39.163268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:59701)
0924 11:41:39.163394 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:39.163410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:39.163490 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:39.163737 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:39.163741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:39.164102 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:39.164303 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.164314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.165418 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.165425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.166557 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.166567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.167430 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.167434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.167547 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.168113 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:39.168133 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:39.170574 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 11:41:39.170944 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:39.163300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59701 (local address 127.0.97.62:36591)
0924 11:41:39.163437 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:39.163440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:39.163453 (+ 13us) server_negotiation.cc:407] Connection header received
0924 11:41:39.163590 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:39.163593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:39.163635 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:39.163693 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:39.164453 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.165287 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.166676 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.167287 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.167423 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.170734 (+ 3311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59701: BlockingRecv error: recv got EOF from 127.0.97.3:59701 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:39.178004 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:39.171291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:44579)
0924 11:41:39.171433 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:39.171447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:39.171545 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:39.171853 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:39.171857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:39.172188 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:39.172312 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.172322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.173220 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.173226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.174028 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.174037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.174621 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.174626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.174789 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.175254 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:39.175267 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:39.177711 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:41:39.178148 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:39.171377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44579 (local address 127.0.97.62:36591)
0924 11:41:39.171596 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:39.171604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:39.171620 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:41:39.171663 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:39.171666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:39.171706 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:39.171831 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:39.172429 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.173110 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.174147 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.174470 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.174596 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.177871 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44579: BlockingRecv error: recv got EOF from 127.0.97.3:44579 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260924 11:41:39.178278 9608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:39.309316 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:39.301115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:48199)
0924 11:41:39.301597 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:39.301612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:39.301718 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:39.301998 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:39.302001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:39.302413 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:39.302577 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.302585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.303491 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.303494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.304471 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.304481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.305341 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.305347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.305484 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.306076 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:39.306101 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:39.309005 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":277,"threads_started":1}
W20260924 11:41:39.309402 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:39.301211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48199 (local address 127.0.97.62:36591)
0924 11:41:39.301365 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:39.301369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:39.301684 (+ 315us) server_negotiation.cc:407] Connection header received
0924 11:41:39.301839 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:39.301843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:39.301895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:39.301964 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:39.302681 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.303378 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.304663 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.305189 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.305275 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.309183 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48199: BlockingRecv error: recv got EOF from 127.0.97.1:48199 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:39 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:39.738722 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:39.731579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:58697)
0924 11:41:39.732115 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:39.732129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:39.732241 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:39.732482 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:39.732485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:39.732876 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:39.733054 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.733061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.733898 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.733901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.734766 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:39.734775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.735398 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.735402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:39.735514 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.736007 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:39.736022 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:39.738523 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":351,"threads_started":1}
W20260924 11:41:39.738953 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:39.731665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58697 (local address 127.0.97.62:36591)
0924 11:41:39.731843 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:39.731848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:39.732207 (+ 359us) server_negotiation.cc:407] Connection header received
0924 11:41:39.732321 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:39.732324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:39.732379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:39.732452 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:39.733153 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.733776 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.734877 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:39.735263 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:39.735345 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:39.738687 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58697: BlockingRecv error: recv got EOF from 127.0.97.2:58697 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 11:41:40 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:40 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:40.186442 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:40.178776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:58069)
0924 11:41:40.179276 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:40.179295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:40.179425 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:40.179789 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:40.179796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:40.180428 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:40.180673 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:40.180687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.181601 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.181607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:40.182467 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:40.182476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.183097 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.183103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:40.183269 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:40.183811 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:40.183826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:40.186135 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":263,"threads_started":1}
W20260924 11:41:40.186542 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:40.178848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58069 (local address 127.0.97.62:36591)
0924 11:41:40.179000 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:40.179004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:40.179372 (+ 368us) server_negotiation.cc:407] Connection header received
0924 11:41:40.179564 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:40.179572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:40.179656 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:40.179733 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:40.180849 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.181478 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.182591 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.182947 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.183066 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:40.186312 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58069: BlockingRecv error: recv got EOF from 127.0.97.3:58069 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 11:41:40 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:40 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:40.317512 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:40.310176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:43951)
0924 11:41:40.310666 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:40.310681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:40.310786 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:40.311065 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:40.311069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:40.311448 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:40.311582 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:40.311590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.312527 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.312531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:40.313384 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:40.313395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.314006 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.314010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:40.314106 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:40.314643 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:40.314659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:40.317173 (+ 2514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":286,"threads_started":1}
W20260924 11:41:40.317579 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:40.310230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43951 (local address 127.0.97.62:36591)
0924 11:41:40.310389 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:40.310394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:40.310750 (+ 356us) server_negotiation.cc:407] Connection header received
0924 11:41:40.310883 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:40.310886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:40.310941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:40.311017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:40.311692 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.312427 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.313490 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.313872 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.313981 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:40.317357 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43951: BlockingRecv error: recv got EOF from 127.0.97.1:43951 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:41:40 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:40 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:40.747236 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:40.739502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:50455)
0924 11:41:40.739951 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:40.739966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:40.740100 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:40.740456 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:40.740460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:40.740862 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:40.740990 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:40.740998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.741945 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.741952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:40.742883 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:40.742894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.743629 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.743635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:40.743798 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:40.744544 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:40.744560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:40.746991 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
W20260924 11:41:40.747376 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:40.739557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50455 (local address 127.0.97.62:36591)
0924 11:41:40.739768 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:40.739774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:40.740037 (+ 263us) server_negotiation.cc:407] Connection header received
0924 11:41:40.740245 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:40.740251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:40.740338 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:40.740430 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:40.741122 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.741827 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.743008 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:40.743487 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:40.743600 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:40.747167 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50455: BlockingRecv error: recv got EOF from 127.0.97.2:50455 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 11:41:41 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:41 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:41.195181 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:41.187313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:47741)
0924 11:41:41.187812 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:41.187827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:41.187969 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:41.188366 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:41.188370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:41.188800 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:41.188933 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:41.188941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.189939 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.189944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:41.190767 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:41.190777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.191478 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.191484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:41.191634 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:41.192185 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:41.192200 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:41.194881 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":254,"threads_started":1}
W20260924 11:41:41.195271 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:41.187473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47741 (local address 127.0.97.62:36591)
0924 11:41:41.187692 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:41.187700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:41.187906 (+ 206us) server_negotiation.cc:407] Connection header received
0924 11:41:41.188086 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:41.188091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:41.188206 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:41.188346 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:41.189076 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.189804 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.190927 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.191341 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.191467 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:41.195039 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47741: BlockingRecv error: recv got EOF from 127.0.97.3:47741 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 11:41:41 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:41 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:41.326135 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:41.318501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:51617)
0924 11:41:41.318959 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:41.318974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:41.319082 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:41.319364 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:41.319367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:41.319689 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:41.319843 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:41.319868 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.321037 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.321042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:41.322060 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:41.322071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.322678 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.322683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:41.322843 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:41.323384 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:41.323399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:41.325915 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
W20260924 11:41:41.326267 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:41.318577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51617 (local address 127.0.97.62:36591)
0924 11:41:41.318835 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:41.318841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:41.319053 (+ 212us) server_negotiation.cc:407] Connection header received
0924 11:41:41.319181 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:41.319185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:41.319241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:41.319324 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:41.319979 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.320893 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.322184 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.322549 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.322666 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:41.326066 (+ 3400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51617: BlockingRecv error: recv got EOF from 127.0.97.1:51617 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 11:41:41 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:41 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:41.755760 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:41.748113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:42895)
0924 11:41:41.748611 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:41.748626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:41.748770 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:41.749080 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:41.749087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:41.749428 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:41.749637 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:41.749646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.750621 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.750626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:41.751499 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:41.751509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.752182 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.752188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:41.752353 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:41.752912 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:41.752928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:41.755415 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":238,"threads_started":1}
W20260924 11:41:41.755864 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:41.748228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42895 (local address 127.0.97.62:36591)
0924 11:41:41.748405 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:41.748412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:41.748702 (+ 290us) server_negotiation.cc:407] Connection header received
0924 11:41:41.748889 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:41.748893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:41.748948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:41.749044 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:41.749782 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.750488 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.751633 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:41.752024 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:41.752140 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:41.755621 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42895: BlockingRecv error: recv got EOF from 127.0.97.2:42895 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 11:41:42 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:42 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:42.204131 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:42.196122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:46957)
0924 11:41:42.196651 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:42.196666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:42.196832 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:42.197160 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:42.197165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:42.197506 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:42.197754 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:42.197766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.199096 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.199100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:42.199982 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:42.199992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.200669 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.200672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:42.200825 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:42.201340 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:42.201357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:42.203811 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":285,"threads_started":1}
W20260924 11:41:42.204206 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:42.196226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46957 (local address 127.0.97.62:36591)
0924 11:41:42.196477 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:42.196485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:42.196757 (+ 272us) server_negotiation.cc:407] Connection header received
0924 11:41:42.196960 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:42.196966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:42.197041 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:42.197139 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:42.197915 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.198989 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.200090 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.200564 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.200651 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:42.203970 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46957: BlockingRecv error: recv got EOF from 127.0.97.3:46957 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 11:41:42 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:42 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:42.334583 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:42.327092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:50655)
0924 11:41:42.327587 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:42.327602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:42.327704 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:42.327962 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:42.327965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:42.328319 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:42.328462 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:42.328470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.329401 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.329404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:42.330336 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:42.330346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.331001 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.331005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:42.331121 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:42.331735 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:42.331751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:42.334275 (+ 2524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":274,"threads_started":1}
W20260924 11:41:42.334736 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:42.327157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50655 (local address 127.0.97.62:36591)
0924 11:41:42.327317 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:42.327322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:42.327666 (+ 344us) server_negotiation.cc:407] Connection header received
0924 11:41:42.327784 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:42.327787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:42.327848 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:42.327923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:42.328570 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.329292 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.330438 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.330879 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.330991 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:42.334479 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50655: BlockingRecv error: recv got EOF from 127.0.97.1:50655 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 11:41:42 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:42 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:42.764405 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:42.756642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:51577)
0924 11:41:42.757202 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:42.757220 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:42.757351 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:42.757615 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:42.757618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:42.757924 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:42.758062 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:42.758070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.758961 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.758964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:42.759891 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:42.759901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.760646 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.760651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:42.760817 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:42.761486 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:42.761510 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:42.764117 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
W20260924 11:41:42.764523 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:42.756724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51577 (local address 127.0.97.62:36591)
0924 11:41:42.756871 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:42.756876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:42.757295 (+ 419us) server_negotiation.cc:407] Connection header received
0924 11:41:42.757442 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:42.757446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:42.757502 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:42.757578 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:42.758164 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.758864 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.760003 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:42.760515 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:42.760616 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:42.764315 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51577: BlockingRecv error: recv got EOF from 127.0.97.2:51577 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 11:41:43 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:43 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:43.213054 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:43.204998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:49501)
0924 11:41:43.205484 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:43.205499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:43.205637 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:43.205940 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:43.205944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:43.206292 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:43.206429 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:43.206438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.207486 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.207493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:43.208580 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:43.208591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.209258 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.209263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:43.209444 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:43.209981 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:43.209997 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:43.212840 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":240,"threads_started":1}
W20260924 11:41:43.213218 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:43.205037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49501 (local address 127.0.97.62:36591)
0924 11:41:43.205237 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:43.205244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:43.205573 (+ 329us) server_negotiation.cc:407] Connection header received
0924 11:41:43.205746 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:43.205752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:43.205814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:43.205916 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:43.206556 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.207354 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.208710 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.209124 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.209247 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:43.212996 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49501: BlockingRecv error: recv got EOF from 127.0.97.3:49501 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 11:41:43 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:43 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:43.343339 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:43.335494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:43413)
0924 11:41:43.335959 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:43.335975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:43.336133 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:43.336464 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:43.336469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:43.336840 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:43.336991 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:43.337000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.338047 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.338054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:43.339085 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:43.339097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.340049 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.340059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:43.340235 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:43.340762 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:43.340778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:43.343124 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":218,"threads_started":1}
W20260924 11:41:43.343466 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:43.335552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43413 (local address 127.0.97.62:36591)
0924 11:41:43.335782 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:43.335788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:43.336060 (+ 272us) server_negotiation.cc:407] Connection header received
0924 11:41:43.336288 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:43.336292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:43.336348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:43.336438 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:43.337124 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.337866 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.339217 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.339907 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.340031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:43.343295 (+ 3264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43413: BlockingRecv error: recv got EOF from 127.0.97.1:43413 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 11:41:43 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:43 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:43.783090 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:43.765242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:50625)
0924 11:41:43.765711 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:43.765725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:43.765822 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:43.766110 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:43.766113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:43.766459 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:43.766622 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:43.766630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.767594 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.767598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:43.768502 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:43.768512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.769125 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.769129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:43.769237 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:43.780490 (+ 11253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:43.780507 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:43.782881 (+ 2374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
W20260924 11:41:43.783236 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:43.765387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50625 (local address 127.0.97.62:36591)
0924 11:41:43.765558 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:43.765562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:43.765787 (+ 225us) server_negotiation.cc:407] Connection header received
0924 11:41:43.765939 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:43.765943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:43.765998 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:43.766075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:43.766729 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.767491 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.768614 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:43.769018 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:43.769104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:43.783033 (+ 13929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50625: BlockingRecv error: recv got EOF from 127.0.97.2:50625 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 11:41:44 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:44 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:44.221067 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:44.213942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:58367)
0924 11:41:44.214378 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:44.214394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:44.214490 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:44.214767 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:44.214769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:44.215143 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:44.215292 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:44.215300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.216309 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.216313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:44.217162 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:44.217171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.217769 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.217772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:44.217886 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:44.218402 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:44.218417 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:44.220835 (+ 2418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
W20260924 11:41:44.221206 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:44.214010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58367 (local address 127.0.97.62:36591)
0924 11:41:44.214163 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:44.214167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:44.214457 (+ 290us) server_negotiation.cc:407] Connection header received
0924 11:41:44.214577 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:44.214580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:44.214649 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:44.214726 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:44.215409 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.216189 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.217276 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.217663 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.217751 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:44.220997 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58367: BlockingRecv error: recv got EOF from 127.0.97.3:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 11:41:44 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:44 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:44.351347 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:44.344132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:59339)
0924 11:41:44.344567 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:44.344581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:44.344675 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:44.344916 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:44.344919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:44.345247 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:44.345418 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:44.345426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.346259 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.346262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:44.347140 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:44.347151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.347796 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.347800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:44.347917 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:44.348488 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:44.348504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:44.351062 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":233,"threads_started":1}
W20260924 11:41:44.351467 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:44.344206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59339 (local address 127.0.97.62:36591)
0924 11:41:44.344380 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:44.344388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:44.344646 (+ 258us) server_negotiation.cc:407] Connection header received
0924 11:41:44.344755 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:44.344758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:44.344810 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:44.344884 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:44.345518 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.346160 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.347251 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.347673 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.347759 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:44.351245 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59339: BlockingRecv error: recv got EOF from 127.0.97.1:59339 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 11:41:44 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:44 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:44.791745 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:44.784182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:42195)
0924 11:41:44.784677 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:44.784691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:44.784790 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:44.785040 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:44.785043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:44.785359 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:44.785500 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:44.785507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.786304 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.786307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:44.787128 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:44.787137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.787870 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.787874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:44.788003 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:44.788611 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:44.788626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:44.791540 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":267,"threads_started":1}
W20260924 11:41:44.791925 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:44.784227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42195 (local address 127.0.97.62:36591)
0924 11:41:44.784391 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:44.784395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:44.784752 (+ 357us) server_negotiation.cc:407] Connection header received
0924 11:41:44.784872 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:44.784875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:44.784928 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:44.785005 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:44.785612 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.786207 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.787274 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:44.787751 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:44.787843 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:44.791724 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42195: BlockingRecv error: recv got EOF from 127.0.97.2:42195 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 11:41:45 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:45 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:45.229283 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:45.222038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:55281)
0924 11:41:45.222526 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:45.222540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:45.222654 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:45.222954 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:45.222957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:45.223283 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:45.223418 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:45.223426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.224371 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.224375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:45.225252 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:45.225261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.225872 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.225876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:45.225992 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:45.226552 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:45.226568 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:45.229082 (+ 2514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":292,"threads_started":1}
W20260924 11:41:45.229494 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:45.222073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55281 (local address 127.0.97.62:36591)
0924 11:41:45.222229 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:45.222233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:45.222607 (+ 374us) server_negotiation.cc:407] Connection header received
0924 11:41:45.222747 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:45.222750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:45.222817 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:45.222911 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:45.223522 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.224268 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.225361 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.225758 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.225852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:45.229241 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55281: BlockingRecv error: recv got EOF from 127.0.97.3:55281 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 11:41:45 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:45 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:45.360005 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:45.352174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:58775)
0924 11:41:45.352679 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:45.352694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:45.352830 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:45.353127 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:45.353132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:45.353487 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:45.353634 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:45.353643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.354742 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.354751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:45.355706 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:45.355717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.356458 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.356462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:45.356566 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:45.357070 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:45.357093 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:45.359643 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":205,"threads_started":1}
W20260924 11:41:45.360016 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:45.352224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58775 (local address 127.0.97.62:36591)
0924 11:41:45.352380 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:45.352396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:45.352767 (+ 371us) server_negotiation.cc:407] Connection header received
0924 11:41:45.352948 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:45.352953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:45.353016 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:45.353105 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:45.353773 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.354622 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.355831 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.356303 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.356429 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:45.359814 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58775: BlockingRecv error: recv got EOF from 127.0.97.1:58775 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 11:41:45 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:45 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:45.800633 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:45.792550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:57561)
0924 11:41:45.793031 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:45.793045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:45.793177 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:45.793526 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:45.793533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:45.793894 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:45.794043 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:45.794052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.795040 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.795045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:45.795897 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:45.795907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.796675 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.796680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:45.796844 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:45.797433 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:45.797449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:45.800320 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":194,"threads_started":1}
W20260924 11:41:45.800693 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:45.792627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57561 (local address 127.0.97.62:36591)
0924 11:41:45.792805 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:45.792812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:45.793123 (+ 311us) server_negotiation.cc:407] Connection header received
0924 11:41:45.793316 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:45.793320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:45.793373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:45.793460 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:45.794179 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.794909 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.796028 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:45.796495 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:45.796663 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:45.800492 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57561: BlockingRecv error: recv got EOF from 127.0.97.2:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 11:41:46 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:46 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:46.237876 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:46.230156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:34927)
0924 11:41:46.230795 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:46.230829 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:46.230972 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:46.231272 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:46.231276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:46.231670 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:46.231820 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:46.231828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.232813 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.232819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:46.233685 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:46.233695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.234344 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.234349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:46.234492 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:46.235035 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:46.235051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:46.237558 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":311,"threads_started":1}
W20260924 11:41:46.237959 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:46.230346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34927 (local address 127.0.97.62:36591)
0924 11:41:46.230543 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:46.230547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:46.230916 (+ 369us) server_negotiation.cc:407] Connection header received
0924 11:41:46.231082 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:46.231092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:46.231155 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:46.231249 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:46.231954 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.232682 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.233817 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.234212 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.234331 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:46.237733 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34927: BlockingRecv error: recv got EOF from 127.0.97.3:34927 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 11:41:46 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:46 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:46.368659 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:46.360900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:48815)
0924 11:41:46.361358 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:46.361373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:46.361518 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:46.361819 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:46.361823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:46.362312 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:46.362511 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:46.362521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.363447 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.363453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:46.364328 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:46.364338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.365012 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.365018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:46.365191 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:46.365818 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:46.365834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:46.368334 (+ 2500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
W20260924 11:41:46.368754 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:46.361058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48815 (local address 127.0.97.62:36591)
0924 11:41:46.361246 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:46.361252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:46.361450 (+ 198us) server_negotiation.cc:407] Connection header received
0924 11:41:46.361644 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:46.361648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:46.361704 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:46.361793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:46.362633 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.363312 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.364451 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.364831 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.364981 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:46.368507 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48815: BlockingRecv error: recv got EOF from 127.0.97.1:48815 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 11:41:46 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:46 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:46.809046 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:46.801505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:40519)
0924 11:41:46.801943 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:46.801956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:46.802084 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:46.802400 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:46.802405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:46.802754 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:46.802880 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:46.802888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.803941 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.803946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:46.804773 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:46.804783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.805506 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.805511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:46.805653 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:46.806126 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:46.806140 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:46.808722 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":243,"threads_started":1}
W20260924 11:41:46.809116 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:46.801569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40519 (local address 127.0.97.62:36591)
0924 11:41:46.801804 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:46.801810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:46.802027 (+ 217us) server_negotiation.cc:407] Connection header received
0924 11:41:46.802190 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:46.802195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:46.802258 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:46.802355 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:46.803012 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.803799 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.804928 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:46.805336 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:46.805452 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:46.808903 (+ 3451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40519: BlockingRecv error: recv got EOF from 127.0.97.2:40519 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 11:41:47 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:47 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:47.246946 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:47.238978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:40841)
0924 11:41:47.239480 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:47.239495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:47.239600 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:47.239915 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:47.239918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:47.240340 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:47.240479 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:47.240487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.241469 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.241472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:47.242364 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:47.242375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.243250 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.243256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:47.243466 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:47.244070 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:47.244094 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:47.246705 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":319,"threads_started":1}
W20260924 11:41:47.247126 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:47.239039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40841 (local address 127.0.97.62:36591)
0924 11:41:47.239201 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:47.239205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:47.239569 (+ 364us) server_negotiation.cc:407] Connection header received
0924 11:41:47.239689 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:47.239694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:47.239781 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:47.239868 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:47.240596 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.241352 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.242502 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.243067 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.243193 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:47.246901 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40841: BlockingRecv error: recv got EOF from 127.0.97.3:40841 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:41:47 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:47 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:47.377125 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:47.369654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:39007)
0924 11:41:47.370154 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:47.370168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:47.370329 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:47.370642 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:47.370646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:47.371006 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:47.371170 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:47.371178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.372115 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.372122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:47.373075 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:47.373085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.373723 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.373728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:47.373875 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:47.374328 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:47.374357 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:47.376823 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":259,"threads_started":1}
W20260924 11:41:47.377220 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:47.369658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39007 (local address 127.0.97.62:36591)
0924 11:41:47.369825 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:47.369832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:47.370248 (+ 416us) server_negotiation.cc:407] Connection header received
0924 11:41:47.370454 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:47.370460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:47.370520 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:47.370611 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:47.371304 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.371992 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.373193 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.373597 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.373712 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:47.377014 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39007: BlockingRecv error: recv got EOF from 127.0.97.1:39007 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 11:41:47 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:47 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:47.817250 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:47.809834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:54137)
0924 11:41:47.810363 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:47.810378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:47.810477 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:47.810739 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:47.810742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:47.811066 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:47.811216 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:47.811224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.812135 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.812170 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:47.813126 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:47.813137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.813721 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.813725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:47.813832 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:47.814400 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:47.814415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:47.816933 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
W20260924 11:41:47.817287 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:47.809875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54137 (local address 127.0.97.62:36591)
0924 11:41:47.810014 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:47.810018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:47.810442 (+ 424us) server_negotiation.cc:407] Connection header received
0924 11:41:47.810564 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:47.810568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:47.810622 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:47.810697 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:47.811324 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.812036 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.813237 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:47.813613 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:47.813695 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:47.817089 (+ 3394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54137: BlockingRecv error: recv got EOF from 127.0.97.2:54137 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 11:41:48 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:48 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:48.255445 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:48.247898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:52145)
0924 11:41:48.248399 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:48.248414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:48.248528 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:48.248829 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:48.248831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:48.249198 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:48.249365 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:48.249374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.250387 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.250390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:48.251291 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:48.251301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.251847 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.251851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:48.251948 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:48.252537 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:48.252551 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:48.255096 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":288,"threads_started":1}
W20260924 11:41:48.255510 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:48.247911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52145 (local address 127.0.97.62:36591)
0924 11:41:48.248121 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:48.248130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:48.248504 (+ 374us) server_negotiation.cc:407] Connection header received
0924 11:41:48.248627 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:48.248630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:48.248704 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:48.248798 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:48.249544 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.250282 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.251394 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.251736 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.251824 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:48.255291 (+ 3467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52145: BlockingRecv error: recv got EOF from 127.0.97.3:52145 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 11:41:48 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:48 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:48.385430 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:48.378025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:59915)
0924 11:41:48.378471 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:48.378486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:48.378583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:48.378840 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:48.378843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:48.379221 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:48.379364 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:48.379372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.380583 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.380590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:48.381584 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:48.381596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.382184 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.382187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:48.382296 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:48.382799 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:48.382814 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:48.385228 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":258,"threads_started":1}
W20260924 11:41:48.385651 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:48.378095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59915 (local address 127.0.97.62:36591)
0924 11:41:48.378301 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:48.378310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:48.378551 (+ 241us) server_negotiation.cc:407] Connection header received
0924 11:41:48.378671 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:48.378674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:48.378727 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:48.378805 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:48.379475 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.380449 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.381691 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.382068 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.382159 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:48.385397 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59915: BlockingRecv error: recv got EOF from 127.0.97.1:59915 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 11:41:48 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:48 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:48.835518 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:48.818318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:58505)
0924 11:41:48.818982 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:48.818997 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:48.819120 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:48.819362 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:48.819365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:48.819721 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:48.819853 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:48.819861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.820793 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.820796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:48.821714 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:48.821725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.822298 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.822301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:48.822413 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:48.832861 (+ 10448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:48.832877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:48.835314 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":460,"threads_started":1}
W20260924 11:41:48.835700 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:48.818335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58505 (local address 127.0.97.62:36591)
0924 11:41:48.818487 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:48.818494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:48.819070 (+ 576us) server_negotiation.cc:407] Connection header received
0924 11:41:48.819212 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:48.819216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:48.819272 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:48.819348 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:48.819968 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.820671 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.821822 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:48.822185 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:48.822271 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:48.835480 (+ 13209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58505: BlockingRecv error: recv got EOF from 127.0.97.2:58505 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 11:41:49 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:49 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:49.264647 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:49.256434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:37863)
0924 11:41:49.256924 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:49.256940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:49.257097 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:49.257464 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:49.257469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:49.257827 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:49.257956 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:49.257965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.258950 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.258957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:49.259914 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:49.259925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.260724 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.260729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:49.260902 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:49.261526 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:49.261543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:49.264323 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":289,"threads_started":1}
W20260924 11:41:49.264756 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:49.256600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37863 (local address 127.0.97.62:36591)
0924 11:41:49.256805 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:49.256811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:49.257018 (+ 207us) server_negotiation.cc:407] Connection header received
0924 11:41:49.257233 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:49.257241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:49.257332 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:49.257436 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:49.258094 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.258816 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.260051 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.260572 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.260712 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:49.264508 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37863: BlockingRecv error: recv got EOF from 127.0.97.3:37863 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 11:41:49 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:49 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:49.393927 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:49.386233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:46013)
0924 11:41:49.386730 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:49.386746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:49.386862 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:49.387133 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:49.387137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:49.387539 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:49.387704 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:49.387712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.388664 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.388667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:49.389557 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:49.389568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.390232 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.390236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:49.390359 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:49.390890 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:49.390905 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:49.393607 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
W20260924 11:41:49.394145 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:49.386335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46013 (local address 127.0.97.62:36591)
0924 11:41:49.386500 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:49.386504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:49.386825 (+ 321us) server_negotiation.cc:407] Connection header received
0924 11:41:49.386958 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:49.386962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:49.387017 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:49.387094 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:49.387827 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.388528 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.389677 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.390088 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.390190 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:49.393874 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46013: BlockingRecv error: recv got EOF from 127.0.97.1:46013 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:41:49 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:49 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:49.844340 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:49.836372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:53787)
0924 11:41:49.836856 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:49.836872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:49.836974 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:49.837255 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:49.837258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:49.837604 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:49.837755 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:49.837762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.838705 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.838708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:49.839578 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:49.839589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.840269 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.840273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:49.840379 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:49.840889 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:49.840904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:49.843997 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
W20260924 11:41:49.844519 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:49.836495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53787 (local address 127.0.97.62:36591)
0924 11:41:49.836654 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:49.836659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:49.836936 (+ 277us) server_negotiation.cc:407] Connection header received
0924 11:41:49.837065 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:49.837069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:49.837131 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:49.837226 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:49.837890 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.838585 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.839688 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:49.840077 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:49.840228 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:49.844285 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53787: BlockingRecv error: recv got EOF from 127.0.97.2:53787 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 11:41:50 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:50 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:50.273855 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:50.265562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:44203)
0924 11:41:50.266033 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:50.266051 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:50.266174 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:50.266472 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:50.266476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:50.266996 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:50.267204 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:50.267219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.268189 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.268194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:50.269076 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:50.269086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.269700 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.269703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:50.269808 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:50.270408 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:50.270424 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:50.273536 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
W20260924 11:41:50.273944 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:50.265639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44203 (local address 127.0.97.62:36591)
0924 11:41:50.265850 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:50.265856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:50.266133 (+ 277us) server_negotiation.cc:407] Connection header received
0924 11:41:50.266276 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:50.266279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:50.266336 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:50.266408 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:50.267336 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.268029 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.269189 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.269580 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.269667 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:50.273722 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44203: BlockingRecv error: recv got EOF from 127.0.97.3:44203 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 11:41:50 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:50 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:50.402907 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:50.395003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:39177)
0924 11:41:50.395486 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:50.395500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:50.395637 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:50.395942 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:50.395945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:50.396284 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:50.396423 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:50.396431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.397387 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.397390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:50.398531 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:50.398548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.399327 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.399331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:50.399451 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:50.399981 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:50.399997 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:50.402590 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
W20260924 11:41:50.402993 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:50.395053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39177 (local address 127.0.97.62:36591)
0924 11:41:50.395282 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:50.395291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:50.395607 (+ 316us) server_negotiation.cc:407] Connection header received
0924 11:41:50.395743 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:50.395746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:50.395818 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:50.395904 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:50.396530 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.397250 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.398677 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.399207 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.399298 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:50.402771 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39177: BlockingRecv error: recv got EOF from 127.0.97.1:39177 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:41:50 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:50 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:50.853063 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:50.845310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:36537)
0924 11:41:50.845722 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:50.845736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:50.845852 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:50.846129 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:50.846132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:50.846612 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:50.846802 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:50.846810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.847874 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.847880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:50.848769 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:50.848779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.849488 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.849493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:50.849672 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:50.850324 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:50.850339 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:50.852746 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":212,"threads_started":1}
W20260924 11:41:50.853103 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:50.845362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36537 (local address 127.0.97.62:36591)
0924 11:41:50.845564 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:50.845571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:50.845804 (+ 233us) server_negotiation.cc:407] Connection header received
0924 11:41:50.845950 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:50.845954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:50.846018 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:50.846092 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:50.846901 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.847711 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.848895 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:50.849348 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:50.849475 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:50.852903 (+ 3428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36537: BlockingRecv error: recv got EOF from 127.0.97.2:36537 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 11:41:51 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:51 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:51.282858 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:51.274633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:50609)
0924 11:41:51.275139 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:51.275153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:51.275298 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:51.275610 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:51.275615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:51.275995 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:51.276176 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:51.276185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.277543 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.277547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:51.278662 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:51.278675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.279635 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.279639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:51.279761 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:51.280374 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:51.280388 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:51.282673 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":248,"threads_started":1}
W20260924 11:41:51.283162 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:51.274741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50609 (local address 127.0.97.62:36591)
0924 11:41:51.274934 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:51.274941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:51.275237 (+ 296us) server_negotiation.cc:407] Connection header received
0924 11:41:51.275416 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:51.275421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:51.275486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:51.275583 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:51.276306 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.277368 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.278819 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.279529 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.279625 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:51.282841 (+ 3216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50609: BlockingRecv error: recv got EOF from 127.0.97.3:50609 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:41:51 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:51 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:51.411751 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:51.403772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:33317)
0924 11:41:51.404367 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:51.404386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:51.404539 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:51.404957 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:51.404960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:51.405373 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:51.405528 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:51.405536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.406426 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.406429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:51.407410 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:51.407420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.408133 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.408177 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:51.408306 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:51.408864 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:51.408889 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:51.411542 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":256,"threads_started":1}
W20260924 11:41:51.411927 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:51.403905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33317 (local address 127.0.97.62:36591)
0924 11:41:51.404119 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:51.404124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:51.404475 (+ 351us) server_negotiation.cc:407] Connection header received
0924 11:41:51.404730 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:51.404736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:51.404801 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:51.404909 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:51.405642 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.406322 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.407521 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.407994 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.408081 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:51.411693 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33317: BlockingRecv error: recv got EOF from 127.0.97.1:33317 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 11:41:51 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:51 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:51.861508 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:51.853898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:52207)
0924 11:41:51.854415 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:51.854447 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:51.854593 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:51.854946 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:51.854950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:51.855409 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:51.855627 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:51.855639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.856715 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.856719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:51.857620 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:51.857630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.858246 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.858250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:51.858351 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:51.858843 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:51.858859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:51.861298 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
W20260924 11:41:51.861681 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:51.854008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52207 (local address 127.0.97.62:36591)
0924 11:41:51.854166 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:51.854170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:51.854544 (+ 374us) server_negotiation.cc:407] Connection header received
0924 11:41:51.854708 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:51.854712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:51.854795 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:51.854883 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:51.855768 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.856582 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.857730 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:51.858103 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:51.858186 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:51.861465 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52207: BlockingRecv error: recv got EOF from 127.0.97.2:52207 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 11:41:52 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:52 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:52.297988 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:52.283630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:45553)
0924 11:41:52.284039 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:52.284054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:52.284217 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:52.284535 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:52.284540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:52.284951 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:52.285095 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:52.285103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.286125 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.286131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:52.287101 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:52.287116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.287910 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.287916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:52.288095 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:52.295048 (+ 6953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:52.295067 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:52.297762 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":192,"threads_started":1}
W20260924 11:41:52.298141 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:52.283646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45553 (local address 127.0.97.62:36591)
0924 11:41:52.283831 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:52.283839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:52.284135 (+ 296us) server_negotiation.cc:407] Connection header received
0924 11:41:52.284335 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:52.284342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:52.284406 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:52.284508 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:52.285234 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.285982 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.287277 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.287715 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.287846 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:52.297950 (+ 10104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45553: BlockingRecv error: recv got EOF from 127.0.97.3:45553 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 11:41:52 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:52 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:52.420997 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:52.412529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:55981)
0924 11:41:52.412970 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:52.412984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:52.413124 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:52.413431 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:52.413435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:52.413779 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:52.413945 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:52.413953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.414879 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.414884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:52.415717 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:52.415727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.416438 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.416444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:52.416604 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:52.418049 (+ 1445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:52.418065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:52.420632 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
W20260924 11:41:52.421051 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:52.412593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55981 (local address 127.0.97.62:36591)
0924 11:41:52.412861 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:52.412866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:52.413056 (+ 190us) server_negotiation.cc:407] Connection header received
0924 11:41:52.413239 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:52.413246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:52.413305 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:52.413408 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:52.414071 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.414744 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.415843 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.416288 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.416419 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:52.420809 (+ 4390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55981: BlockingRecv error: recv got EOF from 127.0.97.1:55981 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 24 11:41:52 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:52 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:52.870563 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:52.862496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:36103)
0924 11:41:52.862960 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:52.862981 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:52.863112 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:52.863436 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:52.863439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:52.863761 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:52.863927 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:52.863934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.864886 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.864890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:52.865747 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:52.865757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.866376 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.866380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:52.866501 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:52.867715 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:52.867731 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:52.870327 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":255,"threads_started":1}
W20260924 11:41:52.870698 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:52.862557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36103 (local address 127.0.97.62:36591)
0924 11:41:52.862789 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:52.862795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:52.863047 (+ 252us) server_negotiation.cc:407] Connection header received
0924 11:41:52.863247 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:52.863251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:52.863309 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:52.863386 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:52.864036 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.864773 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.865863 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:52.866266 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:52.866353 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:52.870486 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36103: BlockingRecv error: recv got EOF from 127.0.97.2:36103 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 11:41:53 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:53 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:53.307875 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:53.298944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.3:54547)
0924 11:41:53.299453 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:53.299473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:53.299594 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:53.299893 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:53.299896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:53.300337 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:53.300526 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:53.300535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.301626 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.301633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:53.302549 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:53.302560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.303219 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.303224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:53.303402 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:53.304842 (+ 1440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:53.304859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:53.307516 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
W20260924 11:41:53.307984 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:53.299045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54547 (local address 127.0.97.62:36591)
0924 11:41:53.299300 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:53.299307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:53.299556 (+ 249us) server_negotiation.cc:407] Connection header received
0924 11:41:53.299693 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:53.299697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:53.299754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:53.299840 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:53.300649 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.301479 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.302689 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.303081 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.303205 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:53.307751 (+ 4546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54547: BlockingRecv error: recv got EOF from 127.0.97.3:54547 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 11:41:53 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:53 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:53.431232 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:53.422256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.1:53027)
0924 11:41:53.422768 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:53.422782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:53.422923 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:53.423252 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:53.423266 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:53.423689 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:53.423881 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:53.423889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.424923 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.424928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:53.425812 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:53.425822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.426529 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.426534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:53.426701 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:53.428050 (+ 1349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:53.428067 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:53.430892 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":277,"threads_started":1}
W20260924 11:41:53.431305 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:53.422348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53027 (local address 127.0.97.62:36591)
0924 11:41:53.422571 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:53.422580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:53.422859 (+ 279us) server_negotiation.cc:407] Connection header received
0924 11:41:53.423053 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:53.423057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:53.423119 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:53.423224 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:53.424022 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.424789 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.425951 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.426366 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.426519 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:53.431065 (+ 4546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53027: BlockingRecv error: recv got EOF from 127.0.97.1:53027 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 11:41:53 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:41:53 dist-test-slave-qhrh krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:41:53.896062 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:53.871535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36591 (local address 127.0.97.2:46525)
0924 11:41:53.872010 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:53.872024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:53.872224 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:53.872543 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:53.872548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:53.872979 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:53.873160 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:53.873176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.874187 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.874193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:53.875074 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:53.875084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.875793 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.875798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:53.875955 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:53.893123 (+ 17168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:41:53.893144 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:53.895739 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36591: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
W20260924 11:41:53.896155 9347 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:53.871595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46525 (local address 127.0.97.62:36591)
0924 11:41:53.871835 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:53.871843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:53.872101 (+ 258us) server_negotiation.cc:407] Connection header received
0924 11:41:53.872343 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:53.872348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:53.872415 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:53.872512 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:53.873322 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.874039 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.875241 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:53.875642 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:53.875783 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:53.895910 (+ 20127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46525: BlockingRecv error: recv got EOF from 127.0.97.2:46525 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:41:54.157475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9198
I20260924 11:41:54.169028 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:54.158347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.0.97.1:46287)
0924 11:41:54.158834 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:54.158841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:54.159368 (+ 527us) server_negotiation.cc:407] Connection header received
0924 11:41:54.159498 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:54.159505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:54.159736 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:54.159900 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:54.161228 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.162164 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.163368 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.163938 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.164030 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:54.166266 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:54.166275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:54.166288 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:54.166318 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:54.168549 (+ 2231us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:41:54.168788 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47806: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":220,"threads_started":1}
W20260924 11:41:54.169127 9657 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47806: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:41:54.169166 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9198: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:46287: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:41:54.169384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9198
I20260924 11:41:54.185519 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9349
I20260924 11:41:54.196033 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:54.186361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.0.97.2:46435)
0924 11:41:54.186916 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:54.186922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:54.186951 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:41:54.187005 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:54.187011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:54.187230 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:54.187374 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:54.188485 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.189473 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.190541 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.191088 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.191176 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:54.193356 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:54.193363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:54.193374 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:54.193397 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:54.195538 (+ 2141us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:41:54.195763 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44488: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":251,"threads_started":1}
W20260924 11:41:54.196142 9659 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44488: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:41:54.196146 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9349: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46435: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:41:54.196372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9349
I20260924 11:41:54.212114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9480
I20260924 11:41:54.222584 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:54.212883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.0.97.3:36283)
0924 11:41:54.213526 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:54.213532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:54.213555 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:41:54.213604 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:54.213610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:54.213809 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:54.213940 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:54.215087 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.216047 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.217188 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.217699 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.217793 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:54.219954 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:54.219963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:54.219975 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:41:54.220010 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:54.222149 (+ 2139us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:41:54.222369 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52488: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":353,"threads_started":1}
W20260924 11:41:54.222690 9660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52488: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:41:54.222771 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9480: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36283: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:41:54.222887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9480
I20260924 11:41:54.238719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9129
I20260924 11:41:54.239902 9191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:54.347913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9129
2026-09-24T11:41:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20711 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T11:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:54Z Disabled control of system clock
I20260924 11:41:54.416517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34307
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 11:41:54.711088 9665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:54.711349 9665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:54.711419 9665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:54.717877 9665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:54.717973 9665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:54.718007 9665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:54.718050 9665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:54.718086 9665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:54.726428 9665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:54.727449 9665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:54.728996 9665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:54.739997 9671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:54.740065 9670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:54.740041 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:54.740545 9665 server_base.cc:1034] running on GCE node
I20260924 11:41:54.741245 9665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:54.742445 9665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:54.743629 9665 hybrid_clock.cc:648] HybridClock initialized: now 1790250114743593 us; error 29 us; skew 500 ppm
I20260924 11:41:54.744059 9665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:54.745940 9665 webserver.cc:460] Webserver started at http://127.0.97.62:35437/ using document root <none> and password file <none>
I20260924 11:41:54.746585 9665 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:54.746692 9665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:54.747004 9665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:54.749636 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "d5f957a783864daa8fcd5d3c4bc139ac"
format_stamp: "Formatted at 2026-09-24 11:41:54 on dist-test-slave-qhrh"
server_key: "de86dbb011a937362ab79e65f4670bb2"
server_key_iv: "26b4d24be765421807f54e25a2d53913"
server_key_version: "encryptionkey@0"
I20260924 11:41:54.750381 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "d5f957a783864daa8fcd5d3c4bc139ac"
format_stamp: "Formatted at 2026-09-24 11:41:54 on dist-test-slave-qhrh"
server_key: "de86dbb011a937362ab79e65f4670bb2"
server_key_iv: "26b4d24be765421807f54e25a2d53913"
server_key_version: "encryptionkey@0"
I20260924 11:41:54.754820 9665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:41:54.757928 9679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:54.759158 9665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:54.759327 9665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "d5f957a783864daa8fcd5d3c4bc139ac"
format_stamp: "Formatted at 2026-09-24 11:41:54 on dist-test-slave-qhrh"
server_key: "de86dbb011a937362ab79e65f4670bb2"
server_key_iv: "26b4d24be765421807f54e25a2d53913"
server_key_version: "encryptionkey@0"
I20260924 11:41:54.759537 9665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:54.790988 9665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:54.791671 9665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:54.791926 9665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:54.808256 9665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34307
I20260924 11:41:54.808285 9730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34307 every 8 connection(s)
I20260924 11:41:54.809381 9665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:54.811939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9665
I20260924 11:41:54.812207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:54.813057 388 external_mini_cluster.cc:1428] Setting key f4acf19a3b831d1c009db44fde4d2198
I20260924 11:41:54.813197 9731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:54.825734 9731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac: Bootstrap starting.
I20260924 11:41:54.828652 9731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:54.829703 9731 log.cc:824] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:54.830533 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:54.815189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.0.97.62:34307)
0924 11:41:54.816767 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:54.816822 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:54.820674 (+ 3852us) server_negotiation.cc:407] Connection header received
0924 11:41:54.822275 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:54.822294 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:54.822716 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:54.823093 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:54.823643 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.824940 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.826249 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:54.826880 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:54.827084 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:54.828220 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:54.828228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:54.828240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:54.828733 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:54.828876 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:54.828898 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:54.829030 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:54.829247 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 11:41:54.829512 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":394,"threads_started":1}
I20260924 11:41:54.832486 9731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac: No bootstrap required, opened a new log
I20260924 11:41:54.839092 9731 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34307 } }
I20260924 11:41:54.839401 9731 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:54.839504 9731 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f957a783864daa8fcd5d3c4bc139ac, State: Initialized, Role: FOLLOWER
I20260924 11:41:54.839962 9731 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34307 } }
I20260924 11:41:54.840245 9731 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:54.840366 9731 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:54.840509 9731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:54.842576 9731 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34307 } }
I20260924 11:41:54.843091 9731 leader_election.cc:302] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f957a783864daa8fcd5d3c4bc139ac; no voters:
I20260924 11:41:54.843675 9731 leader_election.cc:288] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:54.843763 9736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:54.844398 9736 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [term 1 LEADER]: Becoming Leader. State: Replica: d5f957a783864daa8fcd5d3c4bc139ac, State: Running, Role: LEADER
I20260924 11:41:54.844854 9736 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34307 } }
I20260924 11:41:54.845554 9731 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:54.847558 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f957a783864daa8fcd5d3c4bc139ac. Latest consensus state: current_term: 1 leader_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34307 } } }
I20260924 11:41:54.847584 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f957a783864daa8fcd5d3c4bc139ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34307 } } }
I20260924 11:41:54.847930 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:54.847990 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:54.853120 9742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:54.857595 9742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:54.865504 9742 catalog_manager.cc:1347] Generated new cluster ID: d0459bf7259e491cb858a69e13a37d7b
I20260924 11:41:54.865595 9742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:54.881614 9742 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:54.882701 9742 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:54.888489 9742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f957a783864daa8fcd5d3c4bc139ac: Generated new TSK 0
I20260924 11:41:54.889117 9742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:41:54.902271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34307
--builtin_ntp_servers=127.0.97.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 11:41:55.230336 9755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:55.230598 9755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:55.230680 9755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:55.237545 9755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:55.237636 9755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:55.237744 9755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:55.246349 9755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:55.247485 9755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:55.249248 9755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:55.261649 9760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:55.261868 9763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:55.261685 9761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:55.262817 9755 server_base.cc:1034] running on GCE node
I20260924 11:41:55.263455 9755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:55.264356 9755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:55.265580 9755 hybrid_clock.cc:648] HybridClock initialized: now 1790250115265522 us; error 46 us; skew 500 ppm
I20260924 11:41:55.265874 9755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:55.268002 9755 webserver.cc:460] Webserver started at http://127.0.97.1:32797/ using document root <none> and password file <none>
I20260924 11:41:55.268710 9755 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:55.268862 9755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:55.269201 9755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:55.271910 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "46aade626d564a62b829b6f46c1bb270"
format_stamp: "Formatted at 2026-09-24 11:41:55 on dist-test-slave-qhrh"
server_key: "43a6bf0ea3437bceef01cc22167cb2e7"
server_key_iv: "a0c04a48c03a717c17f81ba07f6528ec"
server_key_version: "encryptionkey@0"
I20260924 11:41:55.272746 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "46aade626d564a62b829b6f46c1bb270"
format_stamp: "Formatted at 2026-09-24 11:41:55 on dist-test-slave-qhrh"
server_key: "43a6bf0ea3437bceef01cc22167cb2e7"
server_key_iv: "a0c04a48c03a717c17f81ba07f6528ec"
server_key_version: "encryptionkey@0"
I20260924 11:41:55.277601 9755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:41:55.280791 9769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:55.281903 9755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260924 11:41:55.282088 9755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "46aade626d564a62b829b6f46c1bb270"
format_stamp: "Formatted at 2026-09-24 11:41:55 on dist-test-slave-qhrh"
server_key: "43a6bf0ea3437bceef01cc22167cb2e7"
server_key_iv: "a0c04a48c03a717c17f81ba07f6528ec"
server_key_version: "encryptionkey@0"
I20260924 11:41:55.282298 9755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:55.292383 9755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:55.293108 9755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:55.293386 9755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:55.294242 9755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:55.295572 9755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:55.295696 9755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:55.295810 9755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:55.295903 9755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:55.328661 9755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42171
I20260924 11:41:55.328703 9881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42171 every 8 connection(s)
I20260924 11:41:55.330011 9755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:55.334476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9755
I20260924 11:41:55.334679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:55.335414 388 external_mini_cluster.cc:1428] Setting key 698c9524896951e4c52be6083c5698cd
I20260924 11:41:55.338245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34307
--builtin_ntp_servers=127.0.97.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 11:41:55.343837 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:55.332492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42567 (local address 127.0.97.62:34307)
0924 11:41:55.332895 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:55.332899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:55.333938 (+ 1039us) server_negotiation.cc:407] Connection header received
0924 11:41:55.336375 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:55.336380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:55.336475 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:55.336615 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:55.337728 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.338550 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.340984 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.341397 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.341551 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:55.343056 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:55.343060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:55.343064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:55.343080 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:55.343117 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:55.343128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:55.343199 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:55.343531 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 11:41:55.343616 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":312,"thread_start_us":226,"threads_started":1}
I20260924 11:41:55.344985 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:55.332987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34307 (local address 127.0.97.1:42567)
0924 11:41:55.333801 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:55.333839 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:55.336065 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:55.336782 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:55.336789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:55.336871 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:55.337563 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:55.337586 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.338693 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.338701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:55.340849 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:55.340864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.341677 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.341685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:55.341983 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:55.342707 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:55.342712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:55.342759 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:55.342863 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:55.343222 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:55.343234 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:55.343245 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:41:55.343406 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:41:55.343758 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":268,"threads_started":1}
I20260924 11:41:55.346398 9882 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34307
I20260924 11:41:55.346740 9882 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:55.347535 9882 heartbeater.cc:511] Master 127.0.97.62:34307 requested a full tablet report, sending...
I20260924 11:41:55.349440 9696 ts_manager.cc:194] Registered new tserver with Master: 46aade626d564a62b829b6f46c1bb270 (127.0.97.1:42171)
I20260924 11:41:55.350790 9696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42567
W20260924 11:41:55.640679 9886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:55.640913 9886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:55.641052 9886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:55.647557 9886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:55.647670 9886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:55.647830 9886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:55.655937 9886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:55.657322 9886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:55.658921 9886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:55.671589 9891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:55.671531 9892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:55.671655 9894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:55.672412 9886 server_base.cc:1034] running on GCE node
I20260924 11:41:55.672914 9886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:55.673743 9886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:55.674948 9886 hybrid_clock.cc:648] HybridClock initialized: now 1790250115674908 us; error 55 us; skew 500 ppm
I20260924 11:41:55.675211 9886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:55.677171 9886 webserver.cc:460] Webserver started at http://127.0.97.2:37655/ using document root <none> and password file <none>
I20260924 11:41:55.677779 9886 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:55.677902 9886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:55.678193 9886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:55.680776 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "b0fec6225b194a9da77fe5539223f1d5"
format_stamp: "Formatted at 2026-09-24 11:41:55 on dist-test-slave-qhrh"
server_key: "5d6cf36040671434ffa32a84086f273f"
server_key_iv: "d38267b5c582d781256ec0635fcaf9ec"
server_key_version: "encryptionkey@0"
I20260924 11:41:55.681471 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b0fec6225b194a9da77fe5539223f1d5"
format_stamp: "Formatted at 2026-09-24 11:41:55 on dist-test-slave-qhrh"
server_key: "5d6cf36040671434ffa32a84086f273f"
server_key_iv: "d38267b5c582d781256ec0635fcaf9ec"
server_key_version: "encryptionkey@0"
I20260924 11:41:55.685905 9886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:41:55.689038 9900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:55.690179 9886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:41:55.690343 9886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "b0fec6225b194a9da77fe5539223f1d5"
format_stamp: "Formatted at 2026-09-24 11:41:55 on dist-test-slave-qhrh"
server_key: "5d6cf36040671434ffa32a84086f273f"
server_key_iv: "d38267b5c582d781256ec0635fcaf9ec"
server_key_version: "encryptionkey@0"
I20260924 11:41:55.690518 9886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:55.713291 9886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:55.713946 9886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:55.714195 9886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:55.714977 9886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:55.716241 9886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:55.716342 9886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:55.716459 9886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:55.716544 9886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:55.746817 9886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43155
I20260924 11:41:55.746948 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43155 every 8 connection(s)
I20260924 11:41:55.748267 9886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:55.750741 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 9886
I20260924 11:41:55.750903 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:55.751427 388 external_mini_cluster.cc:1428] Setting key 7746d94a6a4d3e1ed58900ae22450d15
I20260924 11:41:55.753907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34307
--builtin_ntp_servers=127.0.97.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 11:41:55.759831 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:55.750518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39775 (local address 127.0.97.62:34307)
0924 11:41:55.750670 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:55.750674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:55.752135 (+ 1461us) server_negotiation.cc:407] Connection header received
0924 11:41:55.753997 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:55.754002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:55.754098 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:55.754200 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:55.755102 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.755800 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.757380 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.757779 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.757897 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:55.759063 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:55.759068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:55.759071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:55.759086 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:55.759114 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:55.759124 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:55.759215 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:55.759423 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 11:41:55.759509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:41:55.760962 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:55.751103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34307 (local address 127.0.97.2:39775)
0924 11:41:55.751956 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:55.752002 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:55.753763 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:55.754388 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:55.754395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:55.754509 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:55.754961 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:55.754975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.755950 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.755954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:55.757236 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:55.757247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:55.757971 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:55.757977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:55.758182 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:55.758771 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:55.758774 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:55.758805 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:55.758867 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:55.759214 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:55.759225 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:55.759235 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:41:55.759402 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 11:41:55.759738 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260924 11:41:55.762213 10013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34307
I20260924 11:41:55.762485 10013 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:55.763132 10013 heartbeater.cc:511] Master 127.0.97.62:34307 requested a full tablet report, sending...
I20260924 11:41:55.764432 9696 ts_manager.cc:194] Registered new tserver with Master: b0fec6225b194a9da77fe5539223f1d5 (127.0.97.2:43155)
I20260924 11:41:55.765142 9696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39775
W20260924 11:41:56.052438 10016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:56.052655 10016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:56.052770 10016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:56.059597 10016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:56.059675 10016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:56.059851 10016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:56.068497 10016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:56.069586 10016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:56.071162 10016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:56.083452 10021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:56.083663 10022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:56.084000 10024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:56.084506 10016 server_base.cc:1034] running on GCE node
I20260924 11:41:56.085012 10016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:56.085979 10016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:56.087177 10016 hybrid_clock.cc:648] HybridClock initialized: now 1790250116087105 us; error 61 us; skew 500 ppm
I20260924 11:41:56.087460 10016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:56.089618 10016 webserver.cc:460] Webserver started at http://127.0.97.3:34945/ using document root <none> and password file <none>
I20260924 11:41:56.090248 10016 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:56.090330 10016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:56.090585 10016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:56.093212 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "b45549728fc44c00942c38c8d7db8b9d"
format_stamp: "Formatted at 2026-09-24 11:41:56 on dist-test-slave-qhrh"
server_key: "5b642975bdec478b747ed88be24cefbf"
server_key_iv: "78efff64dc0bafc092b6ece3763a821f"
server_key_version: "encryptionkey@0"
I20260924 11:41:56.094085 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b45549728fc44c00942c38c8d7db8b9d"
format_stamp: "Formatted at 2026-09-24 11:41:56 on dist-test-slave-qhrh"
server_key: "5b642975bdec478b747ed88be24cefbf"
server_key_iv: "78efff64dc0bafc092b6ece3763a821f"
server_key_version: "encryptionkey@0"
I20260924 11:41:56.098968 10016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:41:56.102036 10030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:56.103127 10016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:41:56.103269 10016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "b45549728fc44c00942c38c8d7db8b9d"
format_stamp: "Formatted at 2026-09-24 11:41:56 on dist-test-slave-qhrh"
server_key: "5b642975bdec478b747ed88be24cefbf"
server_key_iv: "78efff64dc0bafc092b6ece3763a821f"
server_key_version: "encryptionkey@0"
I20260924 11:41:56.103443 10016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:56.148587 10016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:56.149493 10016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:56.149832 10016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:56.151019 10016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:56.152823 10016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:56.152951 10016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:56.153057 10016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:56.153153 10016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:56.185467 10016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36379
I20260924 11:41:56.185484 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36379 every 8 connection(s)
I20260924 11:41:56.186904 10016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:56.194401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10016
I20260924 11:41:56.194584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:56.195102 388 external_mini_cluster.cc:1428] Setting key 714e035f97c66da15e54f2a1c866c595
I20260924 11:41:56.199156 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:56.189266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40799 (local address 127.0.97.62:34307)
0924 11:41:56.189427 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:56.189431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:56.190665 (+ 1234us) server_negotiation.cc:407] Connection header received
0924 11:41:56.192791 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:56.192795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:56.192857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:56.192933 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:56.193995 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.194777 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.196686 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.197127 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.197227 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:56.198578 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:56.198582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:56.198585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:56.198600 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:56.198627 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:56.198637 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:56.198693 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:56.198856 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 11:41:56.198914 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:41:56.200048 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:56.189662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34307 (local address 127.0.97.3:40799)
0924 11:41:56.190526 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:56.190559 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:56.192525 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:56.193207 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:56.193214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:56.193294 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:56.193816 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:56.193832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.194919 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.194925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:56.196559 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:41:56.196574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.197334 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.197341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:41:56.197610 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:56.198328 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:56.198330 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:56.198363 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:56.198426 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:56.198703 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:56.198708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:56.198713 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:41:56.198832 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 11:41:56.199096 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":281,"threads_started":1}
I20260924 11:41:56.201258 10143 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34307
I20260924 11:41:56.201504 10143 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:56.202126 10143 heartbeater.cc:511] Master 127.0.97.62:34307 requested a full tablet report, sending...
I20260924 11:41:56.203295 9696 ts_manager.cc:194] Registered new tserver with Master: b45549728fc44c00942c38c8d7db8b9d (127.0.97.3:36379)
I20260924 11:41:56.204067 9696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:40799
I20260924 11:41:56.209894 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:41:56.215651 9884 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:56.214028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.0.97.62:34307)
0924 11:41:56.214171 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:56.214175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:56.214749 (+ 574us) server_negotiation.cc:407] Connection header received
0924 11:41:56.214911 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:56.214915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:56.214988 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:56.215080 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:56.215454 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49910: BlockingRecv error: recv got EOF from 127.0.0.1:49910 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:41:56.216058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9755
I20260924 11:41:56.223122 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:56.217114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.0.97.1:42171)
0924 11:41:56.217572 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:56.217578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:56.217599 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:41:56.217644 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:56.217648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:56.217792 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:56.217930 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:56.218279 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.219220 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.220495 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.220946 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.221086 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:56.221944 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:56.221952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:56.221970 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:56.222418 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:56.222453 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:56.222468 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:56.222583 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:56.222734 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 11:41:56.222839 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":200,"threads_started":1}
I20260924 11:41:56.224054 9877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:56.302583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9755
I20260924 11:41:56.318389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9886
I20260924 11:41:56.325841 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:56.319366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.97.2:43155)
0924 11:41:56.319846 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:56.319853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:56.319877 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:41:56.319935 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:56.319941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:56.320201 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:56.320336 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:56.320734 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.321721 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.322875 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.323377 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.323470 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:56.324564 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:56.324571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:56.324583 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:56.325111 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:56.325152 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:56.325168 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:56.325246 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:56.325376 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 11:41:56.325526 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260924 11:41:56.327008 10008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:56.404006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9886
I20260924 11:41:56.419148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10016
I20260924 11:41:56.426574 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:56.420087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.0.97.3:36379)
0924 11:41:56.420720 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:56.420726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:56.420752 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:41:56.420805 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:56.420810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:56.421001 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:56.421119 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:56.421463 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.422443 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.423569 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:41:56.424054 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:41:56.424258 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:41:56.425229 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:56.425235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:56.425247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:56.425794 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:56.425849 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:56.425864 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:56.425943 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:56.426049 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 11:41:56.426205 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":323,"threads_started":1}
I20260924 11:41:56.427685 10138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:56.503170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10016
I20260924 11:41:56.518046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 9665
I20260924 11:41:56.519193 9726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:56.614194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 9665
2026-09-24T11:41:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2261 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T11:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:56Z Disabled control of system clock
I20260924 11:41:56.679471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36637
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 11:41:56.975142 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:56.975394 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:56.975494 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:56.982249 10162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:41:56.982333 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:56.982367 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:56.982419 10162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:41:56.982457 10162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:41:56.990792 10162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:56.991798 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:56.993342 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:57.004544 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:57.004575 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:57.004498 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:57.005478 10162 server_base.cc:1034] running on GCE node
I20260924 11:41:57.006182 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:57.007467 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:57.008680 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790250117008652 us; error 38 us; skew 500 ppm
I20260924 11:41:57.008999 10162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:57.014002 10162 webserver.cc:460] Webserver started at http://127.0.97.62:43903/ using document root <none> and password file <none>
I20260924 11:41:57.014943 10162 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:57.015059 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:57.015400 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:57.018024 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "a5316612d8e14f69beda9320e0e2432e"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "fb0508e419f3b844c29ff2b7a5d32f9d"
server_key_iv: "382b167f3f88a78fac7bc9044ad52751"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.018682 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "a5316612d8e14f69beda9320e0e2432e"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "fb0508e419f3b844c29ff2b7a5d32f9d"
server_key_iv: "382b167f3f88a78fac7bc9044ad52751"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.023208 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:41:57.026237 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.027419 10162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:41:57.027534 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "a5316612d8e14f69beda9320e0e2432e"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "fb0508e419f3b844c29ff2b7a5d32f9d"
server_key_iv: "382b167f3f88a78fac7bc9044ad52751"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.027683 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:57.032676 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:57.033283 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:57.033499 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:57.050118 10162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36637
I20260924 11:41:57.050140 10227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36637 every 8 connection(s)
I20260924 11:41:57.051349 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:41:57.054809 10228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:41:57.054977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10162
I20260924 11:41:57.055184 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:41:57.055878 388 external_mini_cluster.cc:1428] Setting key d12f22ce33d9926ee8b5d89d8ff905b7
I20260924 11:41:57.063823 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:57.057883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.0.97.62:36637)
0924 11:41:57.058935 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:57.058945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:57.058991 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:41:57.060593 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:57.060612 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:57.061012 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:57.061320 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:57.062346 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:57.062353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:57.062364 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:57.062376 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:57.062432 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:57.062446 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:57.062602 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:57.062719 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 11:41:57.062982 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":426,"threads_started":1}
I20260924 11:41:57.066902 10228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e: Bootstrap starting.
I20260924 11:41:57.069630 10228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e: Neither blocks nor log segments found. Creating new log.
I20260924 11:41:57.070541 10228 log.cc:824] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:41:57.072992 10228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e: No bootstrap required, opened a new log
I20260924 11:41:57.079452 10228 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5316612d8e14f69beda9320e0e2432e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36637 } }
I20260924 11:41:57.079766 10228 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:41:57.079860 10228 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5316612d8e14f69beda9320e0e2432e, State: Initialized, Role: FOLLOWER
I20260924 11:41:57.080427 10228 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5316612d8e14f69beda9320e0e2432e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36637 } }
I20260924 11:41:57.080637 10228 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:41:57.080770 10228 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:41:57.080912 10228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:41:57.082736 10228 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5316612d8e14f69beda9320e0e2432e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36637 } }
I20260924 11:41:57.083158 10228 leader_election.cc:302] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5316612d8e14f69beda9320e0e2432e; no voters:
I20260924 11:41:57.083673 10228 leader_election.cc:288] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:41:57.083756 10233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:41:57.084899 10233 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [term 1 LEADER]: Becoming Leader. State: Replica: a5316612d8e14f69beda9320e0e2432e, State: Running, Role: LEADER
I20260924 11:41:57.085395 10233 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5316612d8e14f69beda9320e0e2432e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36637 } }
I20260924 11:41:57.085598 10228 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:41:57.087930 10235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5316612d8e14f69beda9320e0e2432e. Latest consensus state: current_term: 1 leader_uuid: "a5316612d8e14f69beda9320e0e2432e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5316612d8e14f69beda9320e0e2432e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36637 } } }
I20260924 11:41:57.088001 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5316612d8e14f69beda9320e0e2432e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5316612d8e14f69beda9320e0e2432e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36637 } } }
I20260924 11:41:57.088308 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:57.088312 10235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e [sys.catalog]: This master's current role is: LEADER
I20260924 11:41:57.090163 10241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:41:57.094419 10241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:41:57.101845 10241 catalog_manager.cc:1347] Generated new cluster ID: 879f2b09b9b14f47a2b5999e2c4cfe72
I20260924 11:41:57.101941 10241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:41:57.113260 10241 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:41:57.113709 10241 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:41:57.119904 10241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5316612d8e14f69beda9320e0e2432e: Generated new TSK 0
I20260924 11:41:57.120645 10241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:41:57.133528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36637
--builtin_ntp_servers=127.0.97.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 11:41:57.439352 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:57.439587 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:57.439668 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:57.445917 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:57.446002 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:57.446102 10252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:41:57.454283 10252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:57.455279 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:57.456820 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:57.469336 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:57.469406 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:57.469381 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:57.470273 10252 server_base.cc:1034] running on GCE node
I20260924 11:41:57.470790 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:57.471688 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:57.472898 10252 hybrid_clock.cc:648] HybridClock initialized: now 1790250117472843 us; error 54 us; skew 500 ppm
I20260924 11:41:57.473177 10252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:57.475184 10252 webserver.cc:460] Webserver started at http://127.0.97.1:45447/ using document root <none> and password file <none>
I20260924 11:41:57.475805 10252 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:57.475922 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:57.476248 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:57.480288 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "b102252fb96e4c64b1451dcb0ccb8858"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "0fa0567b909da5a27681caab629d766b"
server_key_iv: "6ff80abfb0d15b2a72ebd857a14089f0"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.480970 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b102252fb96e4c64b1451dcb0ccb8858"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "0fa0567b909da5a27681caab629d766b"
server_key_iv: "6ff80abfb0d15b2a72ebd857a14089f0"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.485334 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:41:57.488893 10266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.490150 10252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:41:57.490295 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "b102252fb96e4c64b1451dcb0ccb8858"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "0fa0567b909da5a27681caab629d766b"
server_key_iv: "6ff80abfb0d15b2a72ebd857a14089f0"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.490489 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:57.495355 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:57.495972 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:57.496227 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:57.496973 10252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:57.498097 10252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:57.498174 10252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.498245 10252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:57.498296 10252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.528695 10252 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35687
I20260924 11:41:57.528769 10378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35687 every 8 connection(s)
I20260924 11:41:57.529850 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:41:57.532086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10252
I20260924 11:41:57.532287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:41:57.532938 388 external_mini_cluster.cc:1428] Setting key 258a7c51bab78f885cabe08148b75c41
I20260924 11:41:57.535682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36637
--builtin_ntp_servers=127.0.97.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 11:41:57.538092 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:57.532376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33157 (local address 127.0.97.62:36637)
0924 11:41:57.532584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:57.532590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:57.533926 (+ 1336us) server_negotiation.cc:407] Connection header received
0924 11:41:57.535980 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:57.535984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:57.536027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:57.536106 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:57.537491 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:57.537496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:57.537499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:57.537513 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:57.537542 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:57.537552 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:57.537636 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:57.537855 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 11:41:57.537898 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 11:41:57.539181 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:57.532796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36637 (local address 127.0.97.1:33157)
0924 11:41:57.533773 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:57.533816 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:57.535625 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:57.536367 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:57.536379 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:57.536502 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:57.537165 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:57.537171 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:57.537219 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:57.537329 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:57.537644 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:57.537656 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:57.537666 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:41:57.537920 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 11:41:57.538126 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":311,"threads_started":1}
I20260924 11:41:57.540500 10379 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36637
I20260924 11:41:57.540829 10379 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:57.541383 10379 heartbeater.cc:511] Master 127.0.97.62:36637 requested a full tablet report, sending...
I20260924 11:41:57.543253 10193 ts_manager.cc:194] Registered new tserver with Master: b102252fb96e4c64b1451dcb0ccb8858 (127.0.97.1:35687)
W20260924 11:41:57.839287 10382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:57.839525 10382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:57.839607 10382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:57.846261 10382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:57.846343 10382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:57.846443 10382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:41:57.854871 10382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:57.855880 10382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:57.857419 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:57.869449 10387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:57.869621 10390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:57.869648 10388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:57.870611 10382 server_base.cc:1034] running on GCE node
I20260924 11:41:57.871136 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:57.871960 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:57.873167 10382 hybrid_clock.cc:648] HybridClock initialized: now 1790250117873102 us; error 56 us; skew 500 ppm
I20260924 11:41:57.873445 10382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:57.875362 10382 webserver.cc:460] Webserver started at http://127.0.97.2:44285/ using document root <none> and password file <none>
I20260924 11:41:57.876073 10382 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:57.876196 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:57.876498 10382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:57.879084 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "7275dd13069d41a5852244a592ae3d82"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "e66026faeccc0ce3f7b6836c0895acb8"
server_key_iv: "5dcf1fbf184a5e6a21ede297e37c5280"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.879809 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7275dd13069d41a5852244a592ae3d82"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "e66026faeccc0ce3f7b6836c0895acb8"
server_key_iv: "5dcf1fbf184a5e6a21ede297e37c5280"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.884244 10382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260924 11:41:57.887363 10396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.888491 10382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:57.888646 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "7275dd13069d41a5852244a592ae3d82"
format_stamp: "Formatted at 2026-09-24 11:41:57 on dist-test-slave-qhrh"
server_key: "e66026faeccc0ce3f7b6836c0895acb8"
server_key_iv: "5dcf1fbf184a5e6a21ede297e37c5280"
server_key_version: "encryptionkey@0"
I20260924 11:41:57.888842 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:57.894153 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:57.894853 10382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:57.895097 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:57.895886 10382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:57.897091 10382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:57.897205 10382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.897330 10382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:57.897424 10382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:57.928470 10382 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35457
I20260924 11:41:57.928539 10508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35457 every 8 connection(s)
I20260924 11:41:57.929751 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:41:57.937008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10382
I20260924 11:41:57.937181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:41:57.937405 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:57.932130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40563 (local address 127.0.97.62:36637)
0924 11:41:57.932333 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:57.932337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:57.933693 (+ 1356us) server_negotiation.cc:407] Connection header received
0924 11:41:57.935276 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:57.935281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:57.935347 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:57.935465 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:57.936675 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:57.936699 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:57.936704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:57.936724 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:57.936758 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:57.936769 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:57.936892 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:57.937075 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 11:41:57.937138 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:41:57.937688 388 external_mini_cluster.cc:1428] Setting key cc4a0cd0c6e626c9dd9ca94622bf8692
I20260924 11:41:57.938271 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:57.932693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36637 (local address 127.0.97.2:40563)
0924 11:41:57.933548 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:57.933580 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:57.935027 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:57.935673 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:57.935681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:57.935764 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:57.936408 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:57.936410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:57.936444 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:57.936520 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:57.936858 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:57.936867 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:57.936874 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:41:57.937140 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 11:41:57.937367 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260924 11:41:57.939563 10509 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36637
I20260924 11:41:57.939823 10509 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:57.940371 10509 heartbeater.cc:511] Master 127.0.97.62:36637 requested a full tablet report, sending...
I20260924 11:41:57.940420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36637
--builtin_ntp_servers=127.0.97.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 11:41:57.941687 10193 ts_manager.cc:194] Registered new tserver with Master: 7275dd13069d41a5852244a592ae3d82 (127.0.97.2:35457)
W20260924 11:41:58.242655 10512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:41:58.242949 10512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:41:58.243089 10512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:41:58.249619 10512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:41:58.249740 10512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:41:58.249909 10512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:41:58.258050 10512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:41:58.259183 10512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:41:58.260864 10512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:41:58.273070 10517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:58.273144 10518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:41:58.273030 10520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:41:58.273947 10512 server_base.cc:1034] running on GCE node
I20260924 11:41:58.274518 10512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:41:58.275300 10512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:41:58.276526 10512 hybrid_clock.cc:648] HybridClock initialized: now 1790250118276494 us; error 47 us; skew 500 ppm
I20260924 11:41:58.276798 10512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:41:58.278800 10512 webserver.cc:460] Webserver started at http://127.0.97.3:38369/ using document root <none> and password file <none>
I20260924 11:41:58.279445 10512 fs_manager.cc:359] Metadata directory not provided
I20260924 11:41:58.279567 10512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:41:58.279857 10512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:41:58.282480 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "c26fdc1833854fcea80cc366a0fbad69"
format_stamp: "Formatted at 2026-09-24 11:41:58 on dist-test-slave-qhrh"
server_key: "eca54aef57e28cb48926058d8d744840"
server_key_iv: "dc28a53482c4579b21851c7a550038f1"
server_key_version: "encryptionkey@0"
I20260924 11:41:58.283186 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c26fdc1833854fcea80cc366a0fbad69"
format_stamp: "Formatted at 2026-09-24 11:41:58 on dist-test-slave-qhrh"
server_key: "eca54aef57e28cb48926058d8d744840"
server_key_iv: "dc28a53482c4579b21851c7a550038f1"
server_key_version: "encryptionkey@0"
I20260924 11:41:58.287992 10512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 11:41:58.291173 10526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:58.292419 10512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:41:58.292567 10512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "c26fdc1833854fcea80cc366a0fbad69"
format_stamp: "Formatted at 2026-09-24 11:41:58 on dist-test-slave-qhrh"
server_key: "eca54aef57e28cb48926058d8d744840"
server_key_iv: "dc28a53482c4579b21851c7a550038f1"
server_key_version: "encryptionkey@0"
I20260924 11:41:58.292779 10512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:41:58.297878 10512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:41:58.298544 10512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:41:58.298774 10512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:41:58.299618 10512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:41:58.300900 10512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:41:58.301012 10512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:58.301137 10512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:41:58.301226 10512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:41:58.331796 10512 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32969
I20260924 11:41:58.331818 10638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32969 every 8 connection(s)
I20260924 11:41:58.333043 10512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:41:58.338363 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10512
I20260924 11:41:58.338514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:41:58.339066 388 external_mini_cluster.cc:1428] Setting key c68f60c57dc8a69ea30c2fa7a75e626a
I20260924 11:41:58.340678 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:58.335261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42527 (local address 127.0.97.62:36637)
0924 11:41:58.335467 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:58.335472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:58.336770 (+ 1298us) server_negotiation.cc:407] Connection header received
0924 11:41:58.338300 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:58.338305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:58.338380 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:58.338484 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:58.339869 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:58.339875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:58.339879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:58.339904 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:58.339938 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:58.339952 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:58.340071 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:58.340382 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 11:41:58.340443 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:41:58.341689 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:58.335779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36637 (local address 127.0.97.3:42527)
0924 11:41:58.336623 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 11:41:58.336661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:41:58.338054 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:41:58.338724 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:41:58.338734 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:41:58.338843 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:41:58.339545 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:41:58.339548 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:41:58.339594 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:41:58.339681 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:41:58.340054 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:41:58.340063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:41:58.340069 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:41:58.340448 (+ 379us) client_negotiation.cc:240] Negotiation successful
0924 11:41:58.340660 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":248,"threads_started":1}
I20260924 11:41:58.342803 10639 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36637
I20260924 11:41:58.343011 10639 heartbeater.cc:464] Registering TS with master...
I20260924 11:41:58.343433 10639 heartbeater.cc:511] Master 127.0.97.62:36637 requested a full tablet report, sending...
I20260924 11:41:58.344468 10193 ts_manager.cc:194] Registered new tserver with Master: c26fdc1833854fcea80cc366a0fbad69 (127.0.97.3:32969)
I20260924 11:41:58.353858 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:41:58.359119 10231 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:41:58.357849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.97.62:36637)
0924 11:41:58.358003 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:58.358008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:58.358426 (+ 418us) server_negotiation.cc:407] Connection header received
0924 11:41:58.358527 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:58.358530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:58.358568 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:58.358636 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:58.358877 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36132: BlockingRecv error: recv got EOF from 127.0.0.1:36132 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:41:58.359781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10252
I20260924 11:41:58.362834 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:58.360676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.0.97.1:35687)
0924 11:41:58.361124 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:58.361130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:58.361150 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:41:58.361214 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:58.361218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:58.361363 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:58.361467 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:58.362293 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:58.362298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:58.362306 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:58.362328 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:58.362362 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:58.362373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:58.362462 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:58.362502 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 11:41:58.362582 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260924 11:41:58.363888 10374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:58.441573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10252
I20260924 11:41:58.457039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10382
I20260924 11:41:58.460625 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:58.457855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.97.2:35457)
0924 11:41:58.458292 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:58.458299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:58.458334 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:41:58.458409 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:58.458415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:58.458632 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:58.458891 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:58.459881 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:58.459888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:58.459899 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:58.459915 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:58.459951 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:58.459963 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:58.460096 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:58.460261 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 11:41:58.460379 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260924 11:41:58.461712 10504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:58.540764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10382
I20260924 11:41:58.556790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10512
I20260924 11:41:58.560694 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:41:58.557827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.0.97.3:32969)
0924 11:41:58.558484 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 11:41:58.558490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:41:58.558518 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:41:58.558576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:41:58.558582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:41:58.558787 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:41:58.558909 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:41:58.559868 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:41:58.559874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:41:58.559887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:41:58.559904 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:41:58.559945 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:41:58.559958 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:41:58.560085 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 11:41:58.560275 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 11:41:58.560396 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":352,"threads_started":1}
I20260924 11:41:58.561882 10634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:58.640656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10512
I20260924 11:41:58.655551 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10162
I20260924 11:41:58.656776 10223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:41:58.752031 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10162
2026-09-24T11:41:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2137 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:41:58 dist-test-slave-qhrh krb5kdc[10657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:41:58 dist-test-slave-qhrh krb5kdc[10657](info): set up 2 sockets
Sep 24 11:41:58 dist-test-slave-qhrh krb5kdc[10657](info): commencing operation
krb5kdc: starting...
W20260924 11:41:59.821404 388 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:41:59 dist-test-slave-qhrh krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:42:00.025561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45447
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:42:00.344758 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:00.345058 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:00.345224 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:00.352089 10670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:42:00.352238 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:00.352363 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:00.352463 10670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:42:00.352543 10670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:42:00.361485 10670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:00.362672 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:00.363894 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:00.375603 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:00.375639 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:00.375674 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:00.376466 10670 server_base.cc:1034] running on GCE node
I20260924 11:42:00.377183 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:00.378435 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:00.379654 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790250120379615 us; error 53 us; skew 500 ppm
I20260924 11:42:00.379966 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:00 dist-test-slave-qhrh krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:00.394876 10670 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:42:00.396245 10670 webserver.cc:460] Webserver started at http://127.0.97.62:46319/ using document root <none> and password file <none>
I20260924 11:42:00.396855 10670 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:00.396944 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:00.397163 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:42:00.399821 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "33481523f7984732901965627dff5ec7"
format_stamp: "Formatted at 2026-09-24 11:42:00 on dist-test-slave-qhrh"
server_key: "bde981d20eac575f9ee2f4f45cebc6be"
server_key_iv: "435d3cf6a0e6ae0118c79790bd01ae57"
server_key_version: "encryptionkey@0"
I20260924 11:42:00.400553 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "33481523f7984732901965627dff5ec7"
format_stamp: "Formatted at 2026-09-24 11:42:00 on dist-test-slave-qhrh"
server_key: "bde981d20eac575f9ee2f4f45cebc6be"
server_key_iv: "435d3cf6a0e6ae0118c79790bd01ae57"
server_key_version: "encryptionkey@0"
I20260924 11:42:00.405010 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:42:00.408027 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:00.409222 10670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:42:00.409385 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "33481523f7984732901965627dff5ec7"
format_stamp: "Formatted at 2026-09-24 11:42:00 on dist-test-slave-qhrh"
server_key: "bde981d20eac575f9ee2f4f45cebc6be"
server_key_iv: "435d3cf6a0e6ae0118c79790bd01ae57"
server_key_version: "encryptionkey@0"
I20260924 11:42:00.409543 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:00.438408 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:00.442408 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:00.442667 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:00.459606 10670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45447
I20260924 11:42:00.459649 10736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45447 every 8 connection(s)
I20260924 11:42:00.460875 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:42:00.464133 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:00.465670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10670
I20260924 11:42:00.465878 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:42:00.466683 388 external_mini_cluster.cc:1428] Setting key 97c3abf824867d75b4c8dede76c1ec94
I20260924 11:42:00.477417 10737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7: Bootstrap starting.
Sep 24 11:42:00 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:00.480969 10737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:00.481988 10737 log.cc:824] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:00.485216 10737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7: No bootstrap required, opened a new log
I20260924 11:42:00.489318 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:00.468631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.0.97.62:45447)
0924 11:42:00.469438 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:00.469448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:00.469488 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:42:00.471007 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:00.471025 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:00.471384 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:00.471684 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:00.473394 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:00.474599 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:00.476249 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:00.476803 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:00.476920 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:00.481360 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:00.481366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:00.481378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:00.481406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:00.485051 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:00.485756 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:00.485764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:00.485771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:00.485979 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:00.486309 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:00.486314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:00.486318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:00.486957 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:00.487264 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:00.487566 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 11:42:00.488004 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":374,"threads_started":1}
I20260924 11:42:00.493649 10737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33481523f7984732901965627dff5ec7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45447 } }
I20260924 11:42:00.493939 10737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:00.494004 10737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33481523f7984732901965627dff5ec7, State: Initialized, Role: FOLLOWER
I20260924 11:42:00.494422 10737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33481523f7984732901965627dff5ec7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45447 } }
I20260924 11:42:00.494594 10737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:42:00.494695 10737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:42:00.494801 10737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:00.496759 10737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33481523f7984732901965627dff5ec7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45447 } }
I20260924 11:42:00.497201 10737 leader_election.cc:302] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33481523f7984732901965627dff5ec7; no voters:
I20260924 11:42:00.497716 10737 leader_election.cc:288] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:42:00.497852 10742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:42:00.498513 10742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 33481523f7984732901965627dff5ec7, State: Running, Role: LEADER
I20260924 11:42:00.499024 10742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33481523f7984732901965627dff5ec7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45447 } }
I20260924 11:42:00.499698 10737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:42:00.501487 10743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33481523f7984732901965627dff5ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33481523f7984732901965627dff5ec7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45447 } } }
I20260924 11:42:00.501770 10743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:42:00.503700 10748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:42:00.502440 10744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33481523f7984732901965627dff5ec7. Latest consensus state: current_term: 1 leader_uuid: "33481523f7984732901965627dff5ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33481523f7984732901965627dff5ec7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45447 } } }
I20260924 11:42:00.503921 10744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:42:00.509634 10748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:42:00.520408 10748 catalog_manager.cc:1347] Generated new cluster ID: ff18ae7d566e4b5f9db31e9bc7f3f8e1
I20260924 11:42:00.520514 10748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:42:00.546700 10748 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:42:00.548236 10748 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:42:00.564862 10748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7: Generated new TSK 0
I20260924 11:42:00.565964 10748 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:42:00.615223 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45447
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:42:00.920929 10763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:00.921178 10763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:00.921317 10763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:00.929368 10763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:00.929489 10763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:00.929678 10763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:42:00.943221 10763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:00.944458 10763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:00.946218 10763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:00.961287 10768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:00.961364 10771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:00.962284 10763 server_base.cc:1034] running on GCE node
W20260924 11:42:00.964716 10769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:00.965493 10763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:00.966706 10763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:00.967970 10763 hybrid_clock.cc:648] HybridClock initialized: now 1790250120967890 us; error 46 us; skew 500 ppm
I20260924 11:42:00.968381 10763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:00 dist-test-slave-qhrh krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:00.972095 10763 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:42:00.973523 10763 webserver.cc:460] Webserver started at http://127.0.97.1:45241/ using document root <none> and password file <none>
I20260924 11:42:00.974164 10763 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:00.974304 10763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:00.974581 10763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:42:00.977377 10763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "f1fd58fe898044f4b33accdc05798b74"
format_stamp: "Formatted at 2026-09-24 11:42:00 on dist-test-slave-qhrh"
server_key: "e66731f905142bf7ec86201d182c61a7"
server_key_iv: "0ce2800fcc3cd9ce435cbaee3cdbb6b5"
server_key_version: "encryptionkey@0"
I20260924 11:42:00.978052 10763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f1fd58fe898044f4b33accdc05798b74"
format_stamp: "Formatted at 2026-09-24 11:42:00 on dist-test-slave-qhrh"
server_key: "e66731f905142bf7ec86201d182c61a7"
server_key_iv: "0ce2800fcc3cd9ce435cbaee3cdbb6b5"
server_key_version: "encryptionkey@0"
I20260924 11:42:00.982693 10763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 11:42:00.985766 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:00.986920 10763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:42:00.987093 10763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "f1fd58fe898044f4b33accdc05798b74"
format_stamp: "Formatted at 2026-09-24 11:42:00 on dist-test-slave-qhrh"
server_key: "e66731f905142bf7ec86201d182c61a7"
server_key_iv: "0ce2800fcc3cd9ce435cbaee3cdbb6b5"
server_key_version: "encryptionkey@0"
I20260924 11:42:00.987278 10763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:01.016008 10763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:01.019829 10763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:01.020112 10763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:01.021018 10763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:42:01.022266 10763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:42:01.022367 10763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.022513 10763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:42:01.022589 10763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.053310 10763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36231
I20260924 11:42:01.053402 10890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36231 every 8 connection(s)
I20260924 11:42:01.054554 10763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:42:01.063453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10763
I20260924 11:42:01.063647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:42:01.064507 388 external_mini_cluster.cc:1428] Setting key cc4d1bd32f3e01ddc6ac0a3732064b8d
Sep 24 11:42:01 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:01.075029 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:01.057156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34711 (local address 127.0.97.62:45447)
0924 11:42:01.057562 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:01.057567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:01.058505 (+ 938us) server_negotiation.cc:407] Connection header received
0924 11:42:01.060300 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:01.060305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:01.060392 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:01.060505 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:01.061889 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.062627 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.064990 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.065476 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.065616 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:01.069508 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:01.069513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:01.069516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:01.069545 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:01.072102 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:01.072849 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:01.072855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:01.072859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:01.072975 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:01.073433 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:01.073437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:01.073441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:01.073700 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:01.073837 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:01.074502 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 11:42:01.074693 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
I20260924 11:42:01.075989 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:01.057715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45447 (local address 127.0.97.1:34711)
0924 11:42:01.058344 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:01.058378 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:01.059986 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:01.060726 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:01.060734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:01.061223 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:42:01.061728 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:01.061743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.062794 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.062798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:01.064830 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:01.064843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.065703 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.065707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:01.065905 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:01.066638 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:42:01.066668 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:42:01.069282 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:42:01.072302 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:01.072310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:01.072326 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:01.072728 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:01.073090 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:01.073095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:01.073099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:01.073304 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:01.073823 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:42:01.073830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:42:01.074173 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 11:42:01.074420 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 11:42:01.074907 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":273,"threads_started":1}
I20260924 11:42:01.077453 10891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45447
I20260924 11:42:01.077805 10891 heartbeater.cc:464] Registering TS with master...
I20260924 11:42:01.078588 10891 heartbeater.cc:511] Master 127.0.97.62:45447 requested a full tablet report, sending...
I20260924 11:42:01.080861 10702 ts_manager.cc:194] Registered new tserver with Master: f1fd58fe898044f4b33accdc05798b74 (127.0.97.1:36231)
I20260924 11:42:01.082960 10702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34711
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:42:01.115214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45447
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:42:01.420835 10897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:01.421092 10897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:01.421247 10897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:01.428175 10897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:01.428265 10897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:01.428371 10897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:42:01.437286 10897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:01.438390 10897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:01.439585 10897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:01.452647 10902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:01.452845 10903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:01.453011 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:01.453946 10897 server_base.cc:1034] running on GCE node
I20260924 11:42:01.454520 10897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:01.455353 10897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:01.456565 10897 hybrid_clock.cc:648] HybridClock initialized: now 1790250121456530 us; error 47 us; skew 500 ppm
I20260924 11:42:01.456881 10897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:01 dist-test-slave-qhrh krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:01.459924 10897 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:42:01.469269 10897 webserver.cc:460] Webserver started at http://127.0.97.2:46439/ using document root <none> and password file <none>
I20260924 11:42:01.469911 10897 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:01.470050 10897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:01.470357 10897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:42:01.473002 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "a8093744cb4940679d716cc9dbcc848c"
format_stamp: "Formatted at 2026-09-24 11:42:01 on dist-test-slave-qhrh"
server_key: "c2f098134c957506e0247f51eab7f612"
server_key_iv: "9c2eecd84d22821df0b33946eb503f72"
server_key_version: "encryptionkey@0"
I20260924 11:42:01.473706 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a8093744cb4940679d716cc9dbcc848c"
format_stamp: "Formatted at 2026-09-24 11:42:01 on dist-test-slave-qhrh"
server_key: "c2f098134c957506e0247f51eab7f612"
server_key_iv: "9c2eecd84d22821df0b33946eb503f72"
server_key_version: "encryptionkey@0"
I20260924 11:42:01.478392 10897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:42:01.481600 10912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.482774 10897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 11:42:01.482918 10897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "a8093744cb4940679d716cc9dbcc848c"
format_stamp: "Formatted at 2026-09-24 11:42:01 on dist-test-slave-qhrh"
server_key: "c2f098134c957506e0247f51eab7f612"
server_key_iv: "9c2eecd84d22821df0b33946eb503f72"
server_key_version: "encryptionkey@0"
I20260924 11:42:01.483091 10897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:01.499245 10897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:01.503233 10897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:01.503477 10897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:01.504330 10897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:42:01.505698 10897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:42:01.505784 10897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.505915 10897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:42:01.505990 10897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.537276 10897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46605
I20260924 11:42:01.537317 11024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46605 every 8 connection(s)
I20260924 11:42:01.538453 10897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:42:01.543377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 10897
I20260924 11:42:01.543545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:42:01.544057 388 external_mini_cluster.cc:1428] Setting key e8dab23966bf5f2cca0e557bc09ddc38
Sep 24 11:42:01 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:01.559158 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:01.540988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47831 (local address 127.0.97.62:45447)
0924 11:42:01.541204 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:01.541209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:01.542174 (+ 965us) server_negotiation.cc:407] Connection header received
0924 11:42:01.544110 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:01.544130 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:01.544230 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:01.544313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:01.546016 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.546744 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.549068 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.549457 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.549549 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:01.553396 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:01.553401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:01.553404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:01.553433 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:01.556468 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:01.557085 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:01.557091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:01.557097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:01.557213 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:01.557623 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:01.557628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:01.557631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:01.557887 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:01.558030 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:01.558638 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 11:42:01.558828 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:42:01.560052 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:01.541418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45447 (local address 127.0.97.2:47831)
0924 11:42:01.542025 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:01.542066 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:01.543853 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:01.544560 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:01.544570 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:01.545198 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:42:01.545845 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:01.545865 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.546894 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.546901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:01.548919 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:01.548935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:01.549674 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:01.549683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:01.549985 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:01.550676 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:42:01.550710 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:42:01.553196 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:42:01.556585 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:01.556593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:01.556607 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:01.556978 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:01.557314 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:01.557318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:01.557321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:01.557513 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:01.558006 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:42:01.558013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:42:01.558326 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 11:42:01.558561 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 11:42:01.558993 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":243,"threads_started":1}
I20260924 11:42:01.561476 11025 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45447
I20260924 11:42:01.561789 11025 heartbeater.cc:464] Registering TS with master...
I20260924 11:42:01.562646 11025 heartbeater.cc:511] Master 127.0.97.62:45447 requested a full tablet report, sending...
I20260924 11:42:01.564313 10702 ts_manager.cc:194] Registered new tserver with Master: a8093744cb4940679d716cc9dbcc848c (127.0.97.2:46605)
I20260924 11:42:01.565531 10702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47831
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:42:01.593474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45447
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:42:01.894984 11030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:01.895233 11030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:01.895394 11030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:01.902468 11030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:01.902580 11030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:01.902742 11030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:42:01.912016 11030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:01.913252 11030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:01.914551 11030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:01.927018 11035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:01.927060 11038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:01.927091 11036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:01.927822 11030 server_base.cc:1034] running on GCE node
I20260924 11:42:01.928393 11030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:01.929183 11030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:01.930369 11030 hybrid_clock.cc:648] HybridClock initialized: now 1790250121930327 us; error 41 us; skew 500 ppm
I20260924 11:42:01.930611 11030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:01 dist-test-slave-qhrh krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:01.933640 11030 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:42:01.935019 11030 webserver.cc:460] Webserver started at http://127.0.97.3:35313/ using document root <none> and password file <none>
I20260924 11:42:01.935680 11030 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:01.935804 11030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:01.936089 11030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:42:01.938833 11030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "81b956727f054a81b662b055b08b08b5"
format_stamp: "Formatted at 2026-09-24 11:42:01 on dist-test-slave-qhrh"
server_key: "891b7a49b35c71de6047927f6de8e5e9"
server_key_iv: "f81fd60573d461aacc2931f5469facac"
server_key_version: "encryptionkey@0"
I20260924 11:42:01.939554 11030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "81b956727f054a81b662b055b08b08b5"
format_stamp: "Formatted at 2026-09-24 11:42:01 on dist-test-slave-qhrh"
server_key: "891b7a49b35c71de6047927f6de8e5e9"
server_key_iv: "f81fd60573d461aacc2931f5469facac"
server_key_version: "encryptionkey@0"
I20260924 11:42:01.944047 11030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 11:42:01.947192 11045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.948426 11030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:42:01.948573 11030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "81b956727f054a81b662b055b08b08b5"
format_stamp: "Formatted at 2026-09-24 11:42:01 on dist-test-slave-qhrh"
server_key: "891b7a49b35c71de6047927f6de8e5e9"
server_key_iv: "f81fd60573d461aacc2931f5469facac"
server_key_version: "encryptionkey@0"
I20260924 11:42:01.948758 11030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:01.966109 11030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:01.969576 11030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:01.969889 11030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:01.970785 11030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:42:01.971964 11030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:42:01.972061 11030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:01.972231 11030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:42:01.972306 11030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:02.005249 11030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38513
I20260924 11:42:02.005292 11157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38513 every 8 connection(s)
I20260924 11:42:02.006505 11030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:42:02.011519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 11030
I20260924 11:42:02.011684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:42:02.012281 388 external_mini_cluster.cc:1428] Setting key a331506399765bf44a6db85547c2cfc3
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:02.025115 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.008842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40801 (local address 127.0.97.62:45447)
0924 11:42:02.009028 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.009034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.010219 (+ 1185us) server_negotiation.cc:407] Connection header received
0924 11:42:02.012488 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.012496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.012608 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.012732 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:02.014394 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.015305 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.016790 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.017232 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.017359 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.020491 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:02.020495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:02.020498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:02.020528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:02.022787 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.023387 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.023390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.023392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.023496 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.023830 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.023833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.023835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.024056 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:02.024225 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.024737 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.024886 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:42:02.026008 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.009362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45447 (local address 127.0.97.3:40801)
0924 11:42:02.010080 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:02.010117 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:02.012212 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:02.012995 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:02.013005 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:02.013559 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:42:02.014237 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.014256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.015434 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.015439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.016657 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.016668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.017440 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.017450 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.017679 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.018224 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:42:02.018248 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:42:02.020303 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:42:02.022910 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:02.022916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:02.022925 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:02.023286 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:02.023578 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:02.023581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:02.023585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:02.023738 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:02.024201 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:42:02.024206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:42:02.024451 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 11:42:02.024646 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 11:42:02.024960 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":306,"threads_started":1}
I20260924 11:42:02.027362 11158 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45447
I20260924 11:42:02.027566 11158 heartbeater.cc:464] Registering TS with master...
I20260924 11:42:02.028194 11158 heartbeater.cc:511] Master 127.0.97.62:45447 requested a full tablet report, sending...
I20260924 11:42:02.029417 10701 ts_manager.cc:194] Registered new tserver with Master: 81b956727f054a81b662b055b08b08b5 (127.0.97.3:38513)
I20260924 11:42:02.030228 10701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40801
I20260924 11:42:02.038898 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:42:02.054528 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.043033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.97.62:45447)
0924 11:42:02.043235 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.043242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.043652 (+ 410us) server_negotiation.cc:407] Connection header received
0924 11:42:02.043767 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.043771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.043823 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.043894 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:02.045145 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.045829 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.047022 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.047514 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.047605 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.049684 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:02.049689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:02.049692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:02.049719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:02.051913 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.052603 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.052624 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.052628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.052777 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.053171 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.053176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.053180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.053455 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:02.053684 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.054028 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.054243 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:42:02.058317 10701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:42:02.064894 10701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:42:02.086632 10891 heartbeater.cc:503] Master 127.0.97.62:45447 was elected leader, sending a full tablet report...
I20260924 11:42:02.086887 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.075991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46605 (local address 127.0.0.1:43960)
0924 11:42:02.077062 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:02.077093 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:02.077220 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:02.077914 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:02.077923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:02.077962 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:02.078412 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.078426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.080824 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.080832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.083928 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.083947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.085945 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.085950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.086142 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.086200 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 11:42:02.086316 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 11:42:02.086488 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":804,"threads_started":1}
I20260924 11:42:02.086884 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.076039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.97.2:46605)
0924 11:42:02.076780 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.076793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.077190 (+ 397us) server_negotiation.cc:407] Connection header received
0924 11:42:02.077405 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.077413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.077673 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.077899 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:02.078601 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.080661 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.084190 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.085769 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.085902 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.086085 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.086375 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.086554 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":401,"threads_started":1}
I20260924 11:42:02.090570 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 5f072cbabe9a411d80c66d356dac3aa8 (DEFAULT_TABLE table=test-table [id=94f774274ff54816b9ee24619fe718fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:42:02.092027 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f072cbabe9a411d80c66d356dac3aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:02.093585 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.077435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38513 (local address 127.0.0.1:55362)
0924 11:42:02.083840 (+ 6405us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:02.083856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:02.083962 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:02.086631 (+ 2669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:02.086636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:02.086665 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:02.087036 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.087048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.088782 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.088790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.090843 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.090856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.092745 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.092751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.092959 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.092981 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:42:02.093111 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 11:42:02.093222 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6258,"mutex_wait_us":23,"thread_start_us":6150,"threads_started":1}
I20260924 11:42:02.093622 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.077692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.0.97.3:38513)
0924 11:42:02.085768 (+ 8076us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.085776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.085809 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:42:02.085903 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.085911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.086145 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.086375 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:02.087251 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.088667 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.091044 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.092291 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.092405 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.092610 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.093150 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.093289 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7888,"thread_start_us":7746,"threads_started":1}
I20260924 11:42:02.095830 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.084843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36231 (local address 127.0.0.1:48084)
0924 11:42:02.085447 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:02.085463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:02.085671 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:02.088162 (+ 2491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:02.088170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:02.088193 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:02.088666 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.088680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.090602 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.090610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.093082 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.093102 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.095082 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.095100 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.095270 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.095302 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:42:02.095377 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 11:42:02.095496 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":22,"thread_start_us":291,"threads_started":1}
I20260924 11:42:02.095918 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.084215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.0.97.1:36231)
0924 11:42:02.084971 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.084981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.085540 (+ 559us) server_negotiation.cc:407] Connection header received
0924 11:42:02.085868 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.085875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.086103 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.086279 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:02.088858 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.090429 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.093260 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.094636 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.094725 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.094919 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.095466 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.095640 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":372,"threads_started":1}
I20260924 11:42:02.096683 11093 tablet_service.cc:1467] Processing CreateTablet for tablet 5f072cbabe9a411d80c66d356dac3aa8 (DEFAULT_TABLE table=test-table [id=94f774274ff54816b9ee24619fe718fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:42:02.097900 11093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f072cbabe9a411d80c66d356dac3aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:02.099399 10826 tablet_service.cc:1467] Processing CreateTablet for tablet 5f072cbabe9a411d80c66d356dac3aa8 (DEFAULT_TABLE table=test-table [id=94f774274ff54816b9ee24619fe718fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:42:02.100755 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f072cbabe9a411d80c66d356dac3aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:02.104254 11176 tablet_bootstrap.cc:492] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Bootstrap starting.
I20260924 11:42:02.107797 11176 tablet_bootstrap.cc:654] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:02.108801 11176 log.cc:824] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:02.111382 11176 tablet_bootstrap.cc:492] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: No bootstrap required, opened a new log
I20260924 11:42:02.111706 11176 ts_tablet_manager.cc:1397] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:42:02.111753 11178 tablet_bootstrap.cc:492] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Bootstrap starting.
I20260924 11:42:02.115190 11178 tablet_bootstrap.cc:654] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:02.116379 11178 log.cc:824] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:02.119246 11178 tablet_bootstrap.cc:492] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: No bootstrap required, opened a new log
I20260924 11:42:02.119500 11178 ts_tablet_manager.cc:1397] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 11:42:02.120232 11179 tablet_bootstrap.cc:492] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Bootstrap starting.
I20260924 11:42:02.122403 11176 raft_consensus.cc:348] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.122809 11176 raft_consensus.cc:374] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:02.122946 11176 raft_consensus.cc:729] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8093744cb4940679d716cc9dbcc848c, State: Initialized, Role: FOLLOWER
I20260924 11:42:02.123138 11179 tablet_bootstrap.cc:654] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:02.123543 11176 consensus_queue.cc:260] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.124362 11179 log.cc:824] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:02.129549 11025 heartbeater.cc:503] Master 127.0.97.62:45447 was elected leader, sending a full tablet report...
I20260924 11:42:02.130142 11176 ts_tablet_manager.cc:1428] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260924 11:42:02.130405 11179 tablet_bootstrap.cc:492] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: No bootstrap required, opened a new log
I20260924 11:42:02.130671 11179 ts_tablet_manager.cc:1397] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260924 11:42:02.132107 11178 raft_consensus.cc:348] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.132540 11178 raft_consensus.cc:374] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:02.132645 11178 raft_consensus.cc:729] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b956727f054a81b662b055b08b08b5, State: Initialized, Role: FOLLOWER
I20260924 11:42:02.133314 11178 consensus_queue.cc:260] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.134328 11158 heartbeater.cc:503] Master 127.0.97.62:45447 was elected leader, sending a full tablet report...
I20260924 11:42:02.134887 11178 ts_tablet_manager.cc:1428] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 11:42:02.139578 11179 raft_consensus.cc:348] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.139914 11179 raft_consensus.cc:374] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:02.140027 11179 raft_consensus.cc:729] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fd58fe898044f4b33accdc05798b74, State: Initialized, Role: FOLLOWER
I20260924 11:42:02.140596 11179 consensus_queue.cc:260] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.142096 11179 ts_tablet_manager.cc:1428] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 11:42:02.196087 11184 raft_consensus.cc:482] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:42:02.196405 11184 raft_consensus.cc:504] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.198042 11184 leader_election.cc:288] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8093744cb4940679d716cc9dbcc848c (127.0.97.2:46605), 81b956727f054a81b662b055b08b08b5 (127.0.97.3:38513)
I20260924 11:42:02.204106 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.198334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37313 (local address 127.0.97.2:46605)
0924 11:42:02.198510 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.198514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.198961 (+ 447us) server_negotiation.cc:407] Connection header received
0924 11:42:02.199290 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.199296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.199365 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.199467 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:02.200016 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.200852 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.202500 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.203255 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.203402 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.203492 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.203744 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.203859 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:42:02.204190 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.198257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46605 (local address 127.0.97.1:37313)
0924 11:42:02.198873 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:02.198891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:02.199086 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:02.199506 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:02.199513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:02.199541 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:02.199868 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.199876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.200965 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.200971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.202357 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.202368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.203447 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.203453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.203602 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.203623 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:42:02.203723 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:42:02.203836 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":300,"threads_started":1}
I20260924 11:42:02.204813 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.199027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34445 (local address 127.0.97.3:38513)
0924 11:42:02.199237 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.199244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.199789 (+ 545us) server_negotiation.cc:407] Connection header received
0924 11:42:02.199979 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.199985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.200057 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.200174 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:02.200674 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.201407 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.203065 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.203787 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.203929 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.204025 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.204440 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.204529 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:42:02.204806 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.198932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38513 (local address 127.0.97.1:34445)
0924 11:42:02.199687 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:02.199700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:02.199840 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:02.200181 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:02.200185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:02.200206 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:02.200512 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.200519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.201558 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.201565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.202902 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:02.202911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.203997 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.204002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:02.204224 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.204267 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 11:42:02.204394 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 11:42:02.204506 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":464,"threads_started":1}
I20260924 11:42:02.205341 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f072cbabe9a411d80c66d356dac3aa8" candidate_uuid: "f1fd58fe898044f4b33accdc05798b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8093744cb4940679d716cc9dbcc848c" is_pre_election: true
I20260924 11:42:02.205868 10980 raft_consensus.cc:2454] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1fd58fe898044f4b33accdc05798b74 in term 0.
I20260924 11:42:02.206012 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f072cbabe9a411d80c66d356dac3aa8" candidate_uuid: "f1fd58fe898044f4b33accdc05798b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b956727f054a81b662b055b08b08b5" is_pre_election: true
I20260924 11:42:02.206544 11113 raft_consensus.cc:2454] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1fd58fe898044f4b33accdc05798b74 in term 0.
I20260924 11:42:02.206542 10779 leader_election.cc:302] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8093744cb4940679d716cc9dbcc848c, f1fd58fe898044f4b33accdc05798b74; no voters:
I20260924 11:42:02.206933 11184 raft_consensus.cc:2781] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:42:02.207108 11184 raft_consensus.cc:482] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:42:02.207224 11184 raft_consensus.cc:3037] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:02.209403 11184 raft_consensus.cc:504] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.209987 11184 leader_election.cc:288] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [CANDIDATE]: Term 1 election: Requested vote from peers a8093744cb4940679d716cc9dbcc848c (127.0.97.2:46605), 81b956727f054a81b662b055b08b08b5 (127.0.97.3:38513)
I20260924 11:42:02.210430 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f072cbabe9a411d80c66d356dac3aa8" candidate_uuid: "f1fd58fe898044f4b33accdc05798b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8093744cb4940679d716cc9dbcc848c"
I20260924 11:42:02.210489 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f072cbabe9a411d80c66d356dac3aa8" candidate_uuid: "f1fd58fe898044f4b33accdc05798b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b956727f054a81b662b055b08b08b5"
I20260924 11:42:02.210649 11113 raft_consensus.cc:3037] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:02.210641 10980 raft_consensus.cc:3037] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:02.213855 10980 raft_consensus.cc:2454] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1fd58fe898044f4b33accdc05798b74 in term 1.
I20260924 11:42:02.213855 11113 raft_consensus.cc:2454] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1fd58fe898044f4b33accdc05798b74 in term 1.
I20260924 11:42:02.214257 10779 leader_election.cc:302] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81b956727f054a81b662b055b08b08b5, f1fd58fe898044f4b33accdc05798b74; no voters:
I20260924 11:42:02.214502 11184 raft_consensus.cc:2781] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:42:02.214924 11184 raft_consensus.cc:686] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 1 LEADER]: Becoming Leader. State: Replica: f1fd58fe898044f4b33accdc05798b74, State: Running, Role: LEADER
I20260924 11:42:02.215361 11184 consensus_queue.cc:237] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } }
I20260924 11:42:02.219655 10701 catalog_manager.cc:5495] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1fd58fe898044f4b33accdc05798b74 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f1fd58fe898044f4b33accdc05798b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1fd58fe898044f4b33accdc05798b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36231 } health_report { overall_health: HEALTHY } } }
I20260924 11:42:02.236255 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.231830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.97.1:36231)
0924 11:42:02.232002 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.232006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.232023 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:42:02.232091 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.232094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.232207 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.232319 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:42:02.232614 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.233248 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.234478 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.234849 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.234931 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.235318 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:42:02.235735 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:42:02.235838 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.235909 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.235960 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:42:02.243427 11113 raft_consensus.cc:1261] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 1 FOLLOWER]: Refusing update from remote peer f1fd58fe898044f4b33accdc05798b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:42:02.243429 10980 raft_consensus.cc:1261] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 1 FOLLOWER]: Refusing update from remote peer f1fd58fe898044f4b33accdc05798b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:42:02.244372 11187 consensus_queue.cc:1035] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8093744cb4940679d716cc9dbcc848c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:42:02.244671 11184 consensus_queue.cc:1035] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b956727f054a81b662b055b08b08b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:42:02.268457 10702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:42:02.268692 10702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:42:02.271621 10702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33481523f7984732901965627dff5ec7: Sending DeleteTablet for 3 replicas of tablet 5f072cbabe9a411d80c66d356dac3aa8
I20260924 11:42:02.272548 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f072cbabe9a411d80c66d356dac3aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:42:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43960
I20260924 11:42:02.272792 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f072cbabe9a411d80c66d356dac3aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:42:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48084
I20260924 11:42:02.272786 11093 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f072cbabe9a411d80c66d356dac3aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:42:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55362
I20260924 11:42:02.274896 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10763
I20260924 11:42:02.305312 11201 tablet_replica.cc:330] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: stopping tablet replica
I20260924 11:42:02.305999 11201 raft_consensus.cc:2229] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:42:02.306715 11201 raft_consensus.cc:2258] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:42:02.311271 11200 tablet_replica.cc:330] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: stopping tablet replica
I20260924 11:42:02.311816 11200 raft_consensus.cc:2229] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:42:02.312474 11200 raft_consensus.cc:2258] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:42:02.313709 11201 ts_tablet_manager.cc:1905] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:42:02.317092 11200 ts_tablet_manager.cc:1905] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:42:02.320564 11201 ts_tablet_manager.cc:1918] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:42:02.321138 11202 tablet_replica.cc:330] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: stopping tablet replica
I20260924 11:42:02.321483 11202 raft_consensus.cc:2229] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:42:02.322171 11202 raft_consensus.cc:2258] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:42:02.323662 11202 ts_tablet_manager.cc:1905] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:42:02.326097 11200 ts_tablet_manager.cc:1918] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:42:02.326251 11200 log.cc:1196] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/5f072cbabe9a411d80c66d356dac3aa8
I20260924 11:42:02.326799 11200 ts_tablet_manager.cc:1939] T 5f072cbabe9a411d80c66d356dac3aa8 P a8093744cb4940679d716cc9dbcc848c: Deleting consensus metadata
I20260924 11:42:02.328562 10686 catalog_manager.cc:4841] TS a8093744cb4940679d716cc9dbcc848c (127.0.97.2:46605): tablet 5f072cbabe9a411d80c66d356dac3aa8 (table test-table [id=94f774274ff54816b9ee24619fe718fb]) successfully deleted
I20260924 11:42:02.322526 11201 log.cc:1196] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/5f072cbabe9a411d80c66d356dac3aa8
I20260924 11:42:02.329608 11201 ts_tablet_manager.cc:1939] T 5f072cbabe9a411d80c66d356dac3aa8 P 81b956727f054a81b662b055b08b08b5: Deleting consensus metadata
I20260924 11:42:02.330826 11202 ts_tablet_manager.cc:1918] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:42:02.331029 11202 log.cc:1196] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/5f072cbabe9a411d80c66d356dac3aa8
I20260924 11:42:02.331224 10686 catalog_manager.cc:4841] TS 81b956727f054a81b662b055b08b08b5 (127.0.97.3:38513): tablet 5f072cbabe9a411d80c66d356dac3aa8 (table test-table [id=94f774274ff54816b9ee24619fe718fb]) successfully deleted
I20260924 11:42:02.331622 11202 ts_tablet_manager.cc:1939] T 5f072cbabe9a411d80c66d356dac3aa8 P f1fd58fe898044f4b33accdc05798b74: Deleting consensus metadata
I20260924 11:42:02.333055 10686 catalog_manager.cc:4841] TS f1fd58fe898044f4b33accdc05798b74 (127.0.97.1:36231): tablet 5f072cbabe9a411d80c66d356dac3aa8 (table test-table [id=94f774274ff54816b9ee24619fe718fb]) successfully deleted
I20260924 11:42:02.334508 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.276642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.0.97.1:36231)
0924 11:42:02.276879 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.276887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.301767 (+ 24880us) server_negotiation.cc:407] Connection header received
0924 11:42:02.302289 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.302297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.302404 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.302494 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:02.306955 (+ 4461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.308540 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.313700 (+ 5160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.314632 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.314794 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.327338 (+ 12544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:02.327349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:02.327368 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:02.327414 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:02.331653 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.332364 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.332381 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.332384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.332539 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.332915 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.332923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.332925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.333326 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:02.333612 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.333898 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.334196 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:42:02.335109 10886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:02.432284 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10763
I20260924 11:42:02.450661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10897
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:42:02.464722 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.451431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.97.2:46605)
0924 11:42:02.451596 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.451600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.451714 (+ 114us) server_negotiation.cc:407] Connection header received
0924 11:42:02.451877 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.451881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.451952 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.452054 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:02.453227 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.453869 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.454937 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.455272 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.455352 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.458748 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:02.458759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:02.458773 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:02.458800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:02.462202 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.462741 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.462750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.462753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.462892 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.463194 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.463198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.463202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.463679 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:02.463940 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.464133 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.464381 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:42:02.465340 11020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:02.550349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10897
I20260924 11:42:02.568287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 11030
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:42:02.582276 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:02.568979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.0.97.3:38513)
0924 11:42:02.569138 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:02.569142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:02.569193 (+ 51us) server_negotiation.cc:407] Connection header received
0924 11:42:02.569358 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:02.569362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:02.569424 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:02.569505 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:02.570663 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.571424 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.572569 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:02.572935 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:02.573020 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:02.577025 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:02.577038 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:02.577053 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:02.577090 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:02.579705 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.580360 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.580366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.580368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.580491 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:02.580773 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:02.580776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:02.580777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:02.581068 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:02.581365 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:02.581712 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:42:02.581908 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:42:02.582820 11153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:02.667128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 11030
I20260924 11:42:02.683056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 10670
I20260924 11:42:02.684007 10732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:02.786190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 10670
2026-09-24T11:42:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4036 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[11210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[11210](info): set up 2 sockets
Sep 24 11:42:02 dist-test-slave-qhrh krb5kdc[11210](info): commencing operation
krb5kdc: starting...
W20260924 11:42:03.832928 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:42:03 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:42:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:42:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 11:42:04.094923 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpreJ74/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/postgres -l logfile start
2026-09-24 11:42:06.445 UTC [11240] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 11:42:06.445 UTC [11240] LOG: listening on IPv4 address "127.0.97.20", port 38479
2026-09-24 11:42:06.448 UTC [11240] LOG: listening on Unix socket "/tmp/.s.PGSQL.38479"
2026-09-24 11:42:06.471 UTC [11243] LOG: database system was shut down at 2026-09-24 11:42:05 UTC
2026-09-24 11:42:06.474 UTC [11240] LOG: database system is ready to accept connections
I20260924 11:42:07.408785 388 mini_postgres.cc:96] Postgres bound to 38479
2026-09-24 11:42:07.425 UTC [11251] FATAL: database "slave" does not exist
127.0.97.20:38479 - accepting connections
I20260924 11:42:07.425926 388 mini_ranger.cc:161] Starting Ranger...
I20260924 11:42:07.448330 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 11:42:07.890798 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 11:42:07.890949 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-admin
2026-09-24 11:42:08,181 [I] DB FLAVOR :POSTGRES
2026-09-24 11:42:08,182 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:42:08,182 [I] Checking connection..
2026-09-24 11:42:08,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 11:42:08,421 [I] Checking connection passed.
2026-09-24 11:42:08,421 [I] --------- Verifying version history table ---------
2026-09-24 11:42:08,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:42:08,670 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:42:08,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:42:08,920 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:42:08,920 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 11:42:08,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 11:42:09.172 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 11:42:09,177 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 11:42:09,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:42:09,434 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 11:42:09,435 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:42:09,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 11:42:09,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 11:42:09,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qhrh.c.gcp-upstream.internal','N') ;"
2026-09-24 11:42:10,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:42:10,338 [I] Table x_portal_user does not exist in database ranger
2026-09-24 11:42:10,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:42:10,587 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 11:42:10,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:42:10,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:42:11,053 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 11:42:11,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 11:42:11.950 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:11.965 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:11.980 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:11.993 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.007 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.026 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.073 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.079 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.092 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.101 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.110 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.123 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.131 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.137 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.143 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12.149 UTC [11423] WARNING: there is no transaction in progress
2026-09-24 11:42:12,715 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 11:42:12,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 11:42:12,952 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 11:42:12,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:42:13,202 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 11:42:13,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:42:13,420 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 11:42:13,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:42:13,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:42:13,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qhrh.c.gcp-upstream.internal';"
2026-09-24 11:42:14,104 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 11:42:14,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 11:42:14,336 [I] DB_PATCHES have already been applied
I20260924 11:42:14.341970 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpreJ74/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 11:42:14.342072 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 11:42:14.355276 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:58469
Sep 24, 2026 11:42:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:42:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:42:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:42:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:42:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:42:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:42:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 11:42:15.466718 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.041s user 0.000s sys 0.033s
Sep 24, 2026 11:42:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:42:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 11:42:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:42:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:42:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 11:42:36.496837 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 11:42:36 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250156, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 11:42:36.654433 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 11:42:36.671789 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 11:42:37.485867 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 11:42:37.485993 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms
I20260924 11:42:37.486130 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:38479
2026-09-24 11:42:37,870 [I] DB FLAVOR :POSTGRES
2026-09-24 11:42:37,870 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:42:37,870 [I] Checking connection
2026-09-24 11:42:37,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 11:42:38,083 [I] connection success
2026-09-24 11:42:38,083 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 11:42:38,084 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 11:42:38,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 11:42:38,301 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 11:42:38,301 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:42:38,301 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 11:42:38,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 11:42:38,541 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 11:42:38.964182 388 mini_ranger_kms.cc:310] Created kms service
I20260924 11:42:39.136758 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 11:42:39.181181 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 11:42:39 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250156, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:42:39.547287 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 11:42:39.547591 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 11:42:39.547649 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 11:42:39.565069 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11603
I20260924 11:42:39.565267 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:36705
Sep 24, 2026 11:42:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:42:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:42:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:42:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:42:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:42:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 11:42:41.771756 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.117s user 0.140s sys 0.258s
I20260924 11:42:41.772017 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 11:42:41.772257 388 mini_ranger_kms.cc:401] 127.0.97.20:36705/kms/v1/keys
Sep 24 11:42:41 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:42:42 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 11:42:42 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 11:42:44 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:42:44.904021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34257
--ranger_config_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36705/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:42:45.256348 11654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:45.256633 11654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:45.256785 11654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:45.263924 11654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:42:45.264036 11654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:45.264079 11654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:45.264201 11654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:42:45.264309 11654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:42:45.273919 11654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:36705/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34257
--ranger_config_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:45.275148 11654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:45.276436 11654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:45.289821 11659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:45.289791 11660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:45.290419 11654 server_base.cc:1034] running on GCE node
W20260924 11:42:45.290699 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:45.291528 11654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:45.292999 11654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:45.294235 11654 hybrid_clock.cc:648] HybridClock initialized: now 1790250165294177 us; error 55 us; skew 500 ppm
I20260924 11:42:45.294596 11654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:45 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:45.297757 11654 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:42:45.299207 11654 webserver.cc:460] Webserver started at http://127.0.97.62:33369/ using document root <none> and password file <none>
I20260924 11:42:45.299835 11654 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:45.299949 11654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:45.300303 11654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:42:45 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:42:45.346652 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "bb5abc52572f4e44bce5490a0665f905"
format_stamp: "Formatted at 2026-09-24 11:42:45 on dist-test-slave-qhrh"
server_key: "12c0667f958b9ce72b15b15a085388c1"
server_key_iv: "32752848711e41d623436f9831e34a63"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:45.347473 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "bb5abc52572f4e44bce5490a0665f905"
format_stamp: "Formatted at 2026-09-24 11:42:45 on dist-test-slave-qhrh"
server_key: "12c0667f958b9ce72b15b15a085388c1"
server_key_iv: "32752848711e41d623436f9831e34a63"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:45.353518 11654 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260924 11:42:45.370107 11669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:45.371606 11654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:42:45.371783 11654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "bb5abc52572f4e44bce5490a0665f905"
format_stamp: "Formatted at 2026-09-24 11:42:45 on dist-test-slave-qhrh"
server_key: "12c0667f958b9ce72b15b15a085388c1"
server_key_iv: "32752848711e41d623436f9831e34a63"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:45.372007 11654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:45.388747 11654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:45.392607 11654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:45.392925 11654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:45.412645 11654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34257
I20260924 11:42:45.412665 11720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34257 every 8 connection(s)
I20260924 11:42:45.413813 11654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:42:45.414553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 11654
I20260924 11:42:45.414736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:42:45.415469 388 external_mini_cluster.cc:1428] Setting key 38ea4c55bfa1b6cd013f9b702279a2eb
I20260924 11:42:45.419361 11721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:45.431005 11721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905: Bootstrap starting.
I20260924 11:42:45.434765 11721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:45.435949 11721 log.cc:824] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:42:45 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:45.438709 11721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905: No bootstrap required, opened a new log
I20260924 11:42:45.446871 11721 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5abc52572f4e44bce5490a0665f905" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 11:42:45.447180 11721 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:45.447209 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:45.417764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.0.97.62:34257)
0924 11:42:45.424713 (+ 6949us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:45.424731 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:45.425304 (+ 573us) server_negotiation.cc:407] Connection header received
0924 11:42:45.427557 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:45.427583 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:45.428056 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:45.428504 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:45.429886 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:45.431670 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:45.433466 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:45.434415 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:45.434593 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:45.439245 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:45.439254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:45.439269 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:45.439303 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:45.443018 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:45.443780 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:45.443787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:45.443797 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:45.443951 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:45.444377 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:45.444382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:45.444386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:45.445023 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:45.445308 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:45.445722 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 11:42:45.446060 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6695,"thread_start_us":6498,"threads_started":1}
I20260924 11:42:45.447295 11721 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5abc52572f4e44bce5490a0665f905, State: Initialized, Role: FOLLOWER
I20260924 11:42:45.448450 11721 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5abc52572f4e44bce5490a0665f905" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 11:42:45.448693 11721 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:42:45.448820 11721 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:42:45.448993 11721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:45.451022 11721 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5abc52572f4e44bce5490a0665f905" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 11:42:45.451442 11721 leader_election.cc:302] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb5abc52572f4e44bce5490a0665f905; no voters:
I20260924 11:42:45.451977 11721 leader_election.cc:288] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:42:45.452167 11726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:42:45.452765 11726 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [term 1 LEADER]: Becoming Leader. State: Replica: bb5abc52572f4e44bce5490a0665f905, State: Running, Role: LEADER
I20260924 11:42:45.453275 11726 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5abc52572f4e44bce5490a0665f905" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 11:42:45.454936 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5abc52572f4e44bce5490a0665f905. Latest consensus state: current_term: 1 leader_uuid: "bb5abc52572f4e44bce5490a0665f905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5abc52572f4e44bce5490a0665f905" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } } }
I20260924 11:42:45.455214 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [sys.catalog]: This master's current role is: LEADER
I20260924 11:42:45.455751 11728 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb5abc52572f4e44bce5490a0665f905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5abc52572f4e44bce5490a0665f905" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } } }
I20260924 11:42:45.456125 11728 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [sys.catalog]: This master's current role is: LEADER
I20260924 11:42:45.466164 11721 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:42:45.472496 11730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:42:45.480238 11730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:42:45.493768 11730 catalog_manager.cc:1347] Generated new cluster ID: a86ea4b16ccf44cfa44d0b40b99dad8c
I20260924 11:42:45.493974 11730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:42:45.509428 11721 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 11:42:45.534785 11730 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:42:45.536826 11730 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:42:45.558508 11730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905: Generated new TSK 0
I20260924 11:42:45.559603 11730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:42:46.257529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:42:46 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 11:42:46.635532 11766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:46.635761 11766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:46.635891 11766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:46.643245 11766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:46.643330 11766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:46.643477 11766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:42:46.654676 11766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:36705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:46.656210 11766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:46.657620 11766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:46.676019 11776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:46.677783 11777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:46.686240 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:46.686957 11766 server_base.cc:1034] running on GCE node
I20260924 11:42:46.687712 11766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:46.691286 11766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:46.692540 11766 hybrid_clock.cc:648] HybridClock initialized: now 1790250166692492 us; error 64 us; skew 500 ppm
I20260924 11:42:46.692896 11766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:46 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:46.707142 11766 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:42:46.708920 11766 webserver.cc:460] Webserver started at http://127.0.97.1:34125/ using document root <none> and password file <none>
I20260924 11:42:46.709616 11766 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:46.709744 11766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:46.710059 11766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:42:46 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:42:46.726783 11766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "1d130581723842d5bd8a84aaaa35b395"
format_stamp: "Formatted at 2026-09-24 11:42:46 on dist-test-slave-qhrh"
server_key: "f08624bac120cd1afff9389f5a6fb8e8"
server_key_iv: "ddc51a7c4cda2b8f7d7791a7d7a74303"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:46.729491 11766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1d130581723842d5bd8a84aaaa35b395"
format_stamp: "Formatted at 2026-09-24 11:42:46 on dist-test-slave-qhrh"
server_key: "f08624bac120cd1afff9389f5a6fb8e8"
server_key_iv: "ddc51a7c4cda2b8f7d7791a7d7a74303"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:46.753639 11766 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.007s sys 0.008s
I20260924 11:42:46.782709 11786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:46.784529 11766 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.002s
I20260924 11:42:46.784772 11766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "1d130581723842d5bd8a84aaaa35b395"
format_stamp: "Formatted at 2026-09-24 11:42:46 on dist-test-slave-qhrh"
server_key: "f08624bac120cd1afff9389f5a6fb8e8"
server_key_iv: "ddc51a7c4cda2b8f7d7791a7d7a74303"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:46.785097 11766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:46.819546 11766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:46.826196 11766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:46.826542 11766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:46.828305 11766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:42:46.856899 11766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:42:46.857070 11766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:46.857265 11766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:42:46.857393 11766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:42:47 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:42:47.425186 11898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38867 every 8 connection(s)
I20260924 11:42:47.425544 11766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38867
I20260924 11:42:47.427387 11766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:42:47.436908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 11766
I20260924 11:42:47.437147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:42:47.438114 388 external_mini_cluster.cc:1428] Setting key daac0e90eb0ae730d5d312b5704592c2
Sep 24 11:42:47 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:47.475449 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:47.434302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40489 (local address 127.0.97.62:34257)
0924 11:42:47.454766 (+ 20464us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:47.454772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:47.454791 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:42:47.454849 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:47.454854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:47.454924 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:47.455043 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:47.457566 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:47.458828 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:47.461353 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:47.462151 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:47.462282 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:47.466853 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:47.466859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:47.466864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:47.466911 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:47.470597 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:47.471540 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:47.471545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:47.471549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:47.471704 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:47.472238 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:47.472243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:47.472245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:47.472583 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:47.472733 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:47.474800 (+ 2067us) server_negotiation.cc:299] Negotiation successful
0924 11:42:47.475008 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20340,"thread_start_us":20833,"threads_started":1}
I20260924 11:42:47.475482 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:47.435395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34257 (local address 127.0.97.1:40489)
0924 11:42:47.447370 (+ 11975us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:47.447421 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:47.449792 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:47.455740 (+ 5948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:47.455751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:47.456555 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:42:47.457376 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:47.457400 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:47.459012 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:47.459020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:47.461170 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:47.461187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:47.462563 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:47.462572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:47.462911 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:47.463736 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:42:47.463769 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:42:47.466591 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:42:47.470766 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:47.470774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:47.470791 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:47.471332 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:47.471837 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:47.471841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:47.471844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:47.472077 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:47.472810 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:42:47.472818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:42:47.473222 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 11:42:47.473535 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 11:42:47.474071 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11667,"thread_start_us":14646,"threads_started":1}
I20260924 11:42:47.477975 11899 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34257
I20260924 11:42:47.478399 11899 heartbeater.cc:464] Registering TS with master...
I20260924 11:42:47.479406 11899 heartbeater.cc:511] Master 127.0.97.62:34257 requested a full tablet report, sending...
I20260924 11:42:47.482437 11686 ts_manager.cc:194] Registered new tserver with Master: 1d130581723842d5bd8a84aaaa35b395 (127.0.97.1:38867)
I20260924 11:42:47.484522 11686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40489
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 24 11:42:47 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260924 11:42:47.567838 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.111s user 0.000s sys 0.016s
I20260924 11:42:47.568852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:42:47.983182 11907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:47.983453 11907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:47.983614 11907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:47.990504 11907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:47.990638 11907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:47.990798 11907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:42:48.000435 11907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:36705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:48.001638 11907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:48.002897 11907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:48.015501 11919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:48.015607 11920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:48.015712 11922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:48.016512 11907 server_base.cc:1034] running on GCE node
I20260924 11:42:48.017102 11907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:48.017884 11907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:48.019068 11907 hybrid_clock.cc:648] HybridClock initialized: now 1790250168019020 us; error 43 us; skew 500 ppm
I20260924 11:42:48.019353 11907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:48.022184 11907 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:42:48.023507 11907 webserver.cc:460] Webserver started at http://127.0.97.2:38025/ using document root <none> and password file <none>
I20260924 11:42:48.024125 11907 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:48.024331 11907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:48.024639 11907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:42:48.039952 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "ff5f7ce8e87c4515bf530409dcaadd16"
format_stamp: "Formatted at 2026-09-24 11:42:48 on dist-test-slave-qhrh"
server_key: "6946468f8c53ae1e6db5744c0c757152"
server_key_iv: "8a801691cd3a05a3afd34e92d30a664a"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:48.040827 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ff5f7ce8e87c4515bf530409dcaadd16"
format_stamp: "Formatted at 2026-09-24 11:42:48 on dist-test-slave-qhrh"
server_key: "6946468f8c53ae1e6db5744c0c757152"
server_key_iv: "8a801691cd3a05a3afd34e92d30a664a"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:48.045935 11907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:42:48.062165 11929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:48.063588 11907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:42:48.063768 11907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "ff5f7ce8e87c4515bf530409dcaadd16"
format_stamp: "Formatted at 2026-09-24 11:42:48 on dist-test-slave-qhrh"
server_key: "6946468f8c53ae1e6db5744c0c757152"
server_key_iv: "8a801691cd3a05a3afd34e92d30a664a"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:48.063953 11907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:48.107960 11907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:48.112294 11907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:48.112574 11907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:48.113560 11907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:42:48.114908 11907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:42:48.114974 11907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:48.115088 11907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:42:48.115136 11907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:48.150713 11907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42619
I20260924 11:42:48.150743 12041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42619 every 8 connection(s)
I20260924 11:42:48.151939 11907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:42:48.159277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 11907
I20260924 11:42:48.159435 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:42:48.159899 388 external_mini_cluster.cc:1428] Setting key 436c6ca5a6798434479f5e66265f5b78
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:48.172629 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.154607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51121 (local address 127.0.97.62:34257)
0924 11:42:48.155133 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.155140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.156051 (+ 911us) server_negotiation.cc:407] Connection header received
0924 11:42:48.158275 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.158281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.158358 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.158481 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:48.159841 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.160741 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.162587 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.162968 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.163090 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.166349 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:48.166356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:48.166363 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:48.166399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:48.169307 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.169959 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.169966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.169969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.170083 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.170518 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.170524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.170527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.170870 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:48.171037 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.171864 (+ 827us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.172112 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260924 11:42:48.173610 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.155139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34257 (local address 127.0.97.2:51121)
0924 11:42:48.155871 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.155914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.158004 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.158675 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.158682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.159071 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:42:48.159690 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.159708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.160890 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.160897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.162449 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.162460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.163200 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.163207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.163423 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.164061 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:42:48.164081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:42:48.166159 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:42:48.169433 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:48.169442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:48.169455 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:48.169847 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:48.170183 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:48.170189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:48.170195 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:48.170424 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:48.171051 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:42:48.171058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:42:48.171428 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.171747 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.172260 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
I20260924 11:42:48.175100 12042 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34257
I20260924 11:42:48.175417 12042 heartbeater.cc:464] Registering TS with master...
I20260924 11:42:48.176246 12042 heartbeater.cc:511] Master 127.0.97.62:34257 requested a full tablet report, sending...
I20260924 11:42:48.177738 11686 ts_manager.cc:194] Registered new tserver with Master: ff5f7ce8e87c4515bf530409dcaadd16 (127.0.97.2:42619)
I20260924 11:42:48.178871 11686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51121
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:42:48.209554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:42:48.488520 11899 heartbeater.cc:503] Master 127.0.97.62:34257 was elected leader, sending a full tablet report...
W20260924 11:42:48.538800 12048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:42:48.539053 12048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:42:48.539142 12048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:42:48.545929 12048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:42:48.546032 12048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:42:48.546172 12048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:42:48.554929 12048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:36705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:42:48.556097 12048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:42:48.557423 12048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:42:48.570544 12056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:48.570691 12053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:42:48.570624 12054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:42:48.571960 12048 server_base.cc:1034] running on GCE node
I20260924 11:42:48.572561 12048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:42:48.573508 12048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:42:48.574754 12048 hybrid_clock.cc:648] HybridClock initialized: now 1790250168574692 us; error 45 us; skew 500 ppm
I20260924 11:42:48.575125 12048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:42:48.578190 12048 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:42:48.579497 12048 webserver.cc:460] Webserver started at http://127.0.97.3:42259/ using document root <none> and password file <none>
I20260924 11:42:48.580111 12048 fs_manager.cc:359] Metadata directory not provided
I20260924 11:42:48.580273 12048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:42:48.580559 12048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:42:48.594440 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "1d88c8de9b044b308cc6718e56b6a9f1"
format_stamp: "Formatted at 2026-09-24 11:42:48 on dist-test-slave-qhrh"
server_key: "be27937539d6cdeb2725070e64b830bc"
server_key_iv: "ed8c25b2b72dd122f6ae22cc1b3e6868"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:48.595253 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1d88c8de9b044b308cc6718e56b6a9f1"
format_stamp: "Formatted at 2026-09-24 11:42:48 on dist-test-slave-qhrh"
server_key: "be27937539d6cdeb2725070e64b830bc"
server_key_iv: "ed8c25b2b72dd122f6ae22cc1b3e6868"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:48.600049 12048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 11:42:48.614979 12063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:48.616356 12048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:42:48.616492 12048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "1d88c8de9b044b308cc6718e56b6a9f1"
format_stamp: "Formatted at 2026-09-24 11:42:48 on dist-test-slave-qhrh"
server_key: "be27937539d6cdeb2725070e64b830bc"
server_key_iv: "ed8c25b2b72dd122f6ae22cc1b3e6868"
server_key_version: "kuduclusterkey@0"
I20260924 11:42:48.616652 12048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:42:48.652064 12048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:42:48.655689 12048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:42:48.655959 12048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:42:48.656821 12048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:42:48.657974 12048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:42:48.658037 12048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:48.658178 12048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:42:48.658699 12048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:42:48.690734 12048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33629
I20260924 11:42:48.690762 12175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33629 every 8 connection(s)
I20260924 11:42:48.691939 12048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:42:48.699824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 12048
I20260924 11:42:48.699982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:42:48.700424 388 external_mini_cluster.cc:1428] Setting key 940db95f13fce7c10d0f2d244e921a96
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:42:48.709681 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.694313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36017 (local address 127.0.97.62:34257)
0924 11:42:48.694773 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.694790 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.695578 (+ 788us) server_negotiation.cc:407] Connection header received
0924 11:42:48.697122 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.697126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.697181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.697267 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:48.698453 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.699097 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.700550 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.701378 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.701513 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.704797 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:48.704801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:48.704805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:48.704832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:48.707012 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.707652 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.707656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.707658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.707764 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.708189 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.708193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.708198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.708428 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:48.708566 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.709109 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.709293 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260924 11:42:48.710551 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.694872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34257 (local address 127.0.97.3:36017)
0924 11:42:48.695437 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.695475 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.696909 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.697439 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.697445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.697817 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:42:48.698320 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.698338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.699214 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.699217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.700428 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.700438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.701583 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.701608 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.701846 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.702493 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:42:48.702515 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:42:48.704605 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:42:48.707142 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:48.707149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:48.707163 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:48.707521 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:48.707867 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:42:48.707872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:42:48.707876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:42:48.708058 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:42:48.708542 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:42:48.708548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:42:48.708804 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.709026 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.709463 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":230,"threads_started":1}
I20260924 11:42:48.711640 12176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34257
I20260924 11:42:48.711869 12176 heartbeater.cc:464] Registering TS with master...
I20260924 11:42:48.712514 12176 heartbeater.cc:511] Master 127.0.97.62:34257 requested a full tablet report, sending...
I20260924 11:42:48.713858 11686 ts_manager.cc:194] Registered new tserver with Master: 1d88c8de9b044b308cc6718e56b6a9f1 (127.0.97.3:33629)
I20260924 11:42:48.714965 11686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36017
I20260924 11:42:48.715737 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:42:48.731518 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.719714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.0.97.62:34257)
0924 11:42:48.719860 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.719868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.720323 (+ 455us) server_negotiation.cc:407] Connection header received
0924 11:42:48.720486 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.720491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.720556 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.720650 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:48.721966 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.723000 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.724465 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.724863 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.724988 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.727155 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:48.727161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:48.727166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:48.727201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:48.729269 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.729930 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.729936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.729939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.730048 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.730367 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.730372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.730375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.730580 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:48.730804 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.731080 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.731259 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 11:42:48.734994 11685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:42:48.741405 11685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:42:48.773088 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.752440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59010)
0924 11:42:48.757488 (+ 5048us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.757525 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.757703 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.766541 (+ 8838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.766545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.766569 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:48.766897 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.766904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.769036 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.769041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.770806 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.770814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.771987 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.771993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.772280 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.772341 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.772486 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.772695 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"mutex_wait_us":49,"thread_start_us":762,"threads_started":1}
I20260924 11:42:48.773329 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.752708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.0.97.1:38867)
0924 11:42:48.765845 (+ 13137us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.765853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.765879 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:42:48.766148 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.766155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.766351 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.767097 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:48.767336 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.768896 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.770939 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.771790 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.771933 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.772325 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.772547 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.772697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12936,"thread_start_us":12753,"threads_started":1}
I20260924 11:42:48.773697 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.754151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.97.2:42619)
0924 11:42:48.764682 (+ 10531us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.764688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.764709 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:42:48.764783 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.764788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.764936 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.765134 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:48.766084 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.767750 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.770513 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.771802 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.772634 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.772812 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.773308 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.773443 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10383,"thread_start_us":10273,"threads_started":1}
I20260924 11:42:48.774236 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.753981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42619 (local address 127.0.0.1:56966)
0924 11:42:48.763987 (+ 10006us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.764006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.764152 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.765108 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.765112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.765137 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:48.765853 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.765868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.767947 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.767954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.770393 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.770405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.772176 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.772186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.772394 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.772426 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.772490 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.773877 (+ 1387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9877,"mutex_wait_us":1556,"thread_start_us":8904,"threads_started":1}
I20260924 11:42:48.774642 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.752904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33629 (local address 127.0.0.1:36696)
0924 11:42:48.762767 (+ 9863us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.762787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.762894 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.764562 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.764570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.764606 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:48.765841 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.765855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.767818 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.767824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.770299 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.770310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.772327 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.772333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.772475 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.772506 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.772579 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.772701 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9713,"mutex_wait_us":117,"thread_start_us":11353,"threads_started":1}
I20260924 11:42:48.774272 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.753552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.0.97.3:33629)
0924 11:42:48.763612 (+ 10060us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.763619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.763650 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:42:48.763741 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.763752 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.764038 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.765241 (+ 1203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:48.766018 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.767662 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.770438 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.771802 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.771935 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.772182 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.772804 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.773602 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9888,"thread_start_us":9724,"threads_started":1}
I20260924 11:42:48.776895 11834 tablet_service.cc:1467] Processing CreateTablet for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 (DEFAULT_TABLE table=test-table [id=996acb84a03840349365bdddc89daf67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:42:48.778347 11834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:48.779192 12111 tablet_service.cc:1467] Processing CreateTablet for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 (DEFAULT_TABLE table=test-table [id=996acb84a03840349365bdddc89daf67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:42:48.779152 11977 tablet_service.cc:1467] Processing CreateTablet for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 (DEFAULT_TABLE table=test-table [id=996acb84a03840349365bdddc89daf67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:42:48.780395 11977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:48.780400 12111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:42:48.791164 12195 tablet_bootstrap.cc:492] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Bootstrap starting.
I20260924 11:42:48.794883 12195 tablet_bootstrap.cc:654] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:48.795804 12195 log.cc:824] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:48.796761 12197 tablet_bootstrap.cc:492] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Bootstrap starting.
I20260924 11:42:48.801299 12196 tablet_bootstrap.cc:492] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Bootstrap starting.
I20260924 11:42:48.802169 12197 tablet_bootstrap.cc:654] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:48.803041 12197 log.cc:824] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:48.805405 12196 tablet_bootstrap.cc:654] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Neither blocks nor log segments found. Creating new log.
I20260924 11:42:48.805912 12197 tablet_bootstrap.cc:492] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: No bootstrap required, opened a new log
I20260924 11:42:48.806150 12197 ts_tablet_manager.cc:1397] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 11:42:48.806751 12196 log.cc:824] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Log is configured to *not* fsync() on all Append() calls
I20260924 11:42:48.809442 12195 tablet_bootstrap.cc:492] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: No bootstrap required, opened a new log
I20260924 11:42:48.809695 12195 ts_tablet_manager.cc:1397] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.012s
I20260924 11:42:48.810006 12196 tablet_bootstrap.cc:492] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: No bootstrap required, opened a new log
I20260924 11:42:48.810256 12196 ts_tablet_manager.cc:1397] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:42:48.813583 12197 raft_consensus.cc:348] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.813930 12197 raft_consensus.cc:374] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:48.814049 12197 raft_consensus.cc:729] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5f7ce8e87c4515bf530409dcaadd16, State: Initialized, Role: FOLLOWER
I20260924 11:42:48.814584 12197 consensus_queue.cc:260] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.815482 12042 heartbeater.cc:503] Master 127.0.97.62:34257 was elected leader, sending a full tablet report...
I20260924 11:42:48.816059 12197 ts_tablet_manager.cc:1428] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260924 11:42:48.820364 12195 raft_consensus.cc:348] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.820731 12195 raft_consensus.cc:374] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:48.820818 12195 raft_consensus.cc:729] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d88c8de9b044b308cc6718e56b6a9f1, State: Initialized, Role: FOLLOWER
I20260924 11:42:48.820689 12196 raft_consensus.cc:348] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.821072 12196 raft_consensus.cc:374] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:42:48.821167 12196 raft_consensus.cc:729] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d130581723842d5bd8a84aaaa35b395, State: Initialized, Role: FOLLOWER
I20260924 11:42:48.821408 12195 consensus_queue.cc:260] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.821700 12196 consensus_queue.cc:260] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.822432 12176 heartbeater.cc:503] Master 127.0.97.62:34257 was elected leader, sending a full tablet report...
I20260924 11:42:48.823029 12195 ts_tablet_manager.cc:1428] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 11:42:48.823225 12196 ts_tablet_manager.cc:1428] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 11:42:48.862984 12203 raft_consensus.cc:482] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:42:48.863246 12203 raft_consensus.cc:504] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.864739 12203 leader_election.cc:288] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d88c8de9b044b308cc6718e56b6a9f1 (127.0.97.3:33629), ff5f7ce8e87c4515bf530409dcaadd16 (127.0.97.2:42619)
I20260924 11:42:48.871609 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.865115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51355 (local address 127.0.97.3:33629)
0924 11:42:48.865343 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.865352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.866114 (+ 762us) server_negotiation.cc:407] Connection header received
0924 11:42:48.866292 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.866297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.866366 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.866457 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:48.867578 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.868364 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.870131 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.870771 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.870899 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.870983 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.871278 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.871382 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:42:48.871681 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.865288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53037 (local address 127.0.97.2:42619)
0924 11:42:48.865555 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.865564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.865944 (+ 380us) server_negotiation.cc:407] Connection header received
0924 11:42:48.866150 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.866154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.866214 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.866303 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:42:48.867365 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.868209 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.870149 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.870764 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.870832 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.870887 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.871310 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.871419 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:42:48.871781 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.865049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33629 (local address 127.0.97.1:51355)
0924 11:42:48.866015 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.866028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.866174 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.866515 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.866520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.866544 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:48.867434 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.867444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.868486 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.868491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.869992 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.870003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.870947 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.870952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.871115 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.871135 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.871257 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.871364 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":21,"thread_start_us":487,"threads_started":1}
I20260924 11:42:48.871832 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.865001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42619 (local address 127.0.97.1:53037)
0924 11:42:48.865845 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:48.865860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:42:48.866015 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:42:48.866318 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:42:48.866322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:42:48.866346 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:42:48.867233 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.867244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.868334 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.868341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.869984 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:42:48.869996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.871028 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.871031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:42:48.871151 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.871180 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:42:48.871289 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 11:42:48.871393 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":58,"thread_start_us":768,"threads_started":1}
I20260924 11:42:48.872886 11997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d76feb9e574bc4b0d1f14ab7c0bea1" candidate_uuid: "1d130581723842d5bd8a84aaaa35b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" is_pre_election: true
I20260924 11:42:48.872936 12131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d76feb9e574bc4b0d1f14ab7c0bea1" candidate_uuid: "1d130581723842d5bd8a84aaaa35b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" is_pre_election: true
I20260924 11:42:48.873320 11997 raft_consensus.cc:2454] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d130581723842d5bd8a84aaaa35b395 in term 0.
I20260924 11:42:48.873332 12131 raft_consensus.cc:2454] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d130581723842d5bd8a84aaaa35b395 in term 0.
I20260924 11:42:48.873891 11789 leader_election.cc:302] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d130581723842d5bd8a84aaaa35b395, 1d88c8de9b044b308cc6718e56b6a9f1; no voters:
I20260924 11:42:48.874257 12203 raft_consensus.cc:2781] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:42:48.874382 12203 raft_consensus.cc:482] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:42:48.874476 12203 raft_consensus.cc:3037] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:48.876677 12203 raft_consensus.cc:504] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.877544 12131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d76feb9e574bc4b0d1f14ab7c0bea1" candidate_uuid: "1d130581723842d5bd8a84aaaa35b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d88c8de9b044b308cc6718e56b6a9f1"
I20260924 11:42:48.877651 12203 leader_election.cc:288] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [CANDIDATE]: Term 1 election: Requested vote from peers 1d88c8de9b044b308cc6718e56b6a9f1 (127.0.97.3:33629), ff5f7ce8e87c4515bf530409dcaadd16 (127.0.97.2:42619)
I20260924 11:42:48.877736 12131 raft_consensus.cc:3037] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:48.878036 11997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d76feb9e574bc4b0d1f14ab7c0bea1" candidate_uuid: "1d130581723842d5bd8a84aaaa35b395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5f7ce8e87c4515bf530409dcaadd16"
I20260924 11:42:48.878207 11997 raft_consensus.cc:3037] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:42:48.880559 12131 raft_consensus.cc:2454] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d130581723842d5bd8a84aaaa35b395 in term 1.
I20260924 11:42:48.880892 11789 leader_election.cc:302] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d130581723842d5bd8a84aaaa35b395, 1d88c8de9b044b308cc6718e56b6a9f1; no voters:
I20260924 11:42:48.880991 11997 raft_consensus.cc:2454] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d130581723842d5bd8a84aaaa35b395 in term 1.
I20260924 11:42:48.881174 12203 raft_consensus.cc:2781] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:42:48.881649 12203 raft_consensus.cc:686] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 1 LEADER]: Becoming Leader. State: Replica: 1d130581723842d5bd8a84aaaa35b395, State: Running, Role: LEADER
I20260924 11:42:48.882057 12203 consensus_queue.cc:237] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } }
I20260924 11:42:48.886015 11686 catalog_manager.cc:5495] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d130581723842d5bd8a84aaaa35b395 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1d130581723842d5bd8a84aaaa35b395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d130581723842d5bd8a84aaaa35b395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } health_report { overall_health: UNKNOWN } } }
W20260924 11:42:48.903951 12043 tablet.cc:2369] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:42:48.924042 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.918949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.0.97.1:38867)
0924 11:42:48.919117 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.919121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.919194 (+ 73us) server_negotiation.cc:407] Connection header received
0924 11:42:48.919385 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.919389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.919454 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.919578 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:42:48.919880 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.920617 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.922329 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.922701 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.922801 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.923191 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:42:48.923637 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:42:48.923735 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.923817 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.923872 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:42:48.931448 11997 raft_consensus.cc:1261] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 1 FOLLOWER]: Refusing update from remote peer 1d130581723842d5bd8a84aaaa35b395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:42:48.931452 12131 raft_consensus.cc:1261] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 1 FOLLOWER]: Refusing update from remote peer 1d130581723842d5bd8a84aaaa35b395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:42:48.932433 12206 consensus_queue.cc:1035] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d88c8de9b044b308cc6718e56b6a9f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:42:48.932782 12203 consensus_queue.cc:1035] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5f7ce8e87c4515bf530409dcaadd16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:42:48.942720 12210 mvcc.cc:205] Tried to move back new op lower bound from 7332864691933421568 to 7332864691744325632. Current Snapshot: MvccSnapshot[applied={T|T < 7332864691933421568}]
W20260924 11:42:48.943653 12177 tablet.cc:2369] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:42:48.943895 12209 mvcc.cc:205] Tried to move back new op lower bound from 7332864691933421568 to 7332864691744325632. Current Snapshot: MvccSnapshot[applied={T|T < 7332864691933421568}]
I20260924 11:42:48.958303 11686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:42:48.958570 11686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:42:48.961724 11686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb5abc52572f4e44bce5490a0665f905: Sending DeleteTablet for 3 replicas of tablet 06d76feb9e574bc4b0d1f14ab7c0bea1
I20260924 11:42:48.962772 11834 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:42:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59010
I20260924 11:42:48.962782 12111 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:42:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36696
I20260924 11:42:48.963097 11977 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:42:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56966
I20260924 11:42:48.965253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 11766
I20260924 11:42:48.973212 12220 tablet_replica.cc:330] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: stopping tablet replica
I20260924 11:42:48.973665 12220 raft_consensus.cc:2229] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:42:48.974184 12220 raft_consensus.cc:2258] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:42:48.975869 12221 tablet_replica.cc:330] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: stopping tablet replica
I20260924 11:42:48.976430 12221 raft_consensus.cc:2229] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:42:48.976926 12221 raft_consensus.cc:2258] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:42:48.979230 12220 ts_tablet_manager.cc:1905] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:42:48.979281 12221 ts_tablet_manager.cc:1905] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:42:48.985821 12220 ts_tablet_manager.cc:1918] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:42:48.986424 12220 log.cc:1196] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/06d76feb9e574bc4b0d1f14ab7c0bea1
I20260924 11:42:48.986897 12220 ts_tablet_manager.cc:1939] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d88c8de9b044b308cc6718e56b6a9f1: Deleting consensus metadata
Sep 24 11:42:48 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:42:48.988798 12219 tablet_replica.cc:330] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: stopping tablet replica
I20260924 11:42:48.988754 11672 catalog_manager.cc:4841] TS 1d88c8de9b044b308cc6718e56b6a9f1 (127.0.97.3:33629): tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 (table test-table [id=996acb84a03840349365bdddc89daf67]) successfully deleted
I20260924 11:42:48.989166 12219 raft_consensus.cc:2229] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:42:48.989269 12221 ts_tablet_manager.cc:1918] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:42:48.989411 12221 log.cc:1196] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/06d76feb9e574bc4b0d1f14ab7c0bea1
I20260924 11:42:48.989880 12221 ts_tablet_manager.cc:1939] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P ff5f7ce8e87c4515bf530409dcaadd16: Deleting consensus metadata
I20260924 11:42:48.989928 12219 raft_consensus.cc:2258] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:42:48.991849 11670 catalog_manager.cc:4841] TS ff5f7ce8e87c4515bf530409dcaadd16 (127.0.97.2:42619): tablet 06d76feb9e574bc4b0d1f14ab7c0bea1 (table test-table [id=996acb84a03840349365bdddc89daf67]) successfully deleted
I20260924 11:42:48.991955 12219 ts_tablet_manager.cc:1905] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:42:48.997817 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:48.967077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.97.1:38867)
0924 11:42:48.967287 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:48.967296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:48.977696 (+ 10400us) server_negotiation.cc:407] Connection header received
0924 11:42:48.977882 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:48.977887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:48.977979 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:48.978090 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:48.979818 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.980967 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.982901 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:48.983536 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:48.983656 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:48.990461 (+ 6805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:48.990476 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:48.990490 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:48.990561 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:48.994654 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.995507 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.995517 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.995521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.995695 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:48.996173 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:48.996178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:48.996182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:48.996627 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:48.996948 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:48.997241 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 11:42:48.997458 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:42:48.998589 11894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:49.095703 12219 ts_tablet_manager.cc:1918] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:42:49.095970 12219 log.cc:1196] T 06d76feb9e574bc4b0d1f14ab7c0bea1 P 1d130581723842d5bd8a84aaaa35b395: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/06d76feb9e574bc4b0d1f14ab7c0bea1
I20260924 11:42:49.096468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 11766
W20260924 11:42:49.113597 11670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38867 (error 108)
I20260924 11:42:49.113814 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 11907
I20260924 11:42:49.115504 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.115060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59036)
0924 11:42:49.115235 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.115343 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:42:49.115763 11670 catalog_manager.cc:4584] TS 1d130581723842d5bd8a84aaaa35b395 (127.0.97.1:38867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06d76feb9e574bc4b0d1f14ab7c0bea1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Sep 24 11:42:49 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:42:49.127866 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.114544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.0.97.2:42619)
0924 11:42:49.114717 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:49.114722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:49.114792 (+ 70us) server_negotiation.cc:407] Connection header received
0924 11:42:49.115032 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:49.115038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:49.115125 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:49.115232 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:49.116610 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:49.117298 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:49.118512 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:49.118857 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:49.118939 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:49.122938 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:49.122949 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:49.122962 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:49.122993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:49.125580 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:49.126230 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:49.126236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:49.126238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:49.126345 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:49.126621 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:49.126624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:49.126626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:49.126981 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:49.127183 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:49.127393 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 11:42:49.127527 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:42:49.128523 12037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:49.169032 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.168591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59040)
0924 11:42:49.168762 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.168863 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:42:49.169960 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.169609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59042)
0924 11:42:49.169758 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.169845 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:42:49.213114 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 11907
I20260924 11:42:49.229960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 12048
I20260924 11:42:49.230453 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.229941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59058)
0924 11:42:49.230173 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.230260 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:42:49.231426 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.231068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59066)
0924 11:42:49.231218 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.231294 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Sep 24 11:42:49 dist-test-slave-qhrh krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:42:49.243659 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.230734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.0.97.3:33629)
0924 11:42:49.230880 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 11:42:49.230884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:42:49.231003 (+ 119us) server_negotiation.cc:407] Connection header received
0924 11:42:49.231198 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:42:49.231203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:42:49.231281 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:42:49.231382 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:42:49.232722 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:49.233418 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:49.234566 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:42:49.234912 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:42:49.234994 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:42:49.238927 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:42:49.238939 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:42:49.238953 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:42:49.238984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:42:49.241539 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:49.242092 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:49.242099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:49.242101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:49.242216 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:42:49.242494 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:42:49.242497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:42:49.242499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:42:49.242846 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:42:49.243040 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 11:42:49.243213 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 11:42:49.243357 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:42:49.244282 12171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:49.311956 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.311532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59082)
0924 11:42:49.311748 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.311822 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:42:49.312947 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:42:49.312586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38867 (local address 127.0.0.1:59086)
0924 11:42:49.312784 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 11:42:49.312852 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 11:42:49.329399 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 12048
I20260924 11:42:49.344789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 11654
I20260924 11:42:49.345868 11716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:42:49.450793 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 11654
I20260924 11:42:49.475958 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 11:42:49.811192 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 11:42:49.811416 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 11:42:50.156076 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 11:42:50.156 UTC [11240] LOG: received smart shutdown request
2026-09-24 11:42:50.159 UTC [11240] LOG: background worker "logical replication launcher" (PID 11249) exited with exit code 1
2026-09-24 11:42:50.159 UTC [11244] LOG: shutting down
2026-09-24 11:42:50.323 UTC [11240] LOG: database system is shut down
2026-09-24T11:42:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47650 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:42:50 dist-test-slave-qhrh krb5kdc[12236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:42:50 dist-test-slave-qhrh krb5kdc[12236](info): set up 2 sockets
Sep 24 11:42:50 dist-test-slave-qhrh krb5kdc[12236](info): commencing operation
krb5kdc: starting...
W20260924 11:42:51.496217 388 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:42:51 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:42:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 11:42:51.766206 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpreJ74/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/postgres -l logfile start
2026-09-24 11:42:54.036 UTC [12266] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 11:42:54.036 UTC [12266] LOG: listening on IPv4 address "127.0.97.20", port 35741
2026-09-24 11:42:54.040 UTC [12266] LOG: listening on Unix socket "/tmp/.s.PGSQL.35741"
2026-09-24 11:42:54.061 UTC [12269] LOG: database system was shut down at 2026-09-24 11:42:52 UTC
2026-09-24 11:42:54.064 UTC [12266] LOG: database system is ready to accept connections
I20260924 11:42:54.999832 388 mini_postgres.cc:96] Postgres bound to 35741
2026-09-24 11:42:55.017 UTC [12277] FATAL: database "slave" does not exist
127.0.97.20:35741 - accepting connections
I20260924 11:42:55.018494 388 mini_ranger.cc:161] Starting Ranger...
I20260924 11:42:55.046550 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 11:42:55.470765 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 11:42:55.470896 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-admin
2026-09-24 11:42:55,750 [I] DB FLAVOR :POSTGRES
2026-09-24 11:42:55,750 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:42:55,750 [I] Checking connection..
2026-09-24 11:42:55,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 11:42:55,962 [I] Checking connection passed.
2026-09-24 11:42:55,962 [I] --------- Verifying version history table ---------
2026-09-24 11:42:55,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:42:56,203 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:42:56,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:42:56,445 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:42:56,445 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 11:42:56,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 11:42:56.668 UTC [12341] WARNING: there is no transaction in progress
2026-09-24 11:42:56,674 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 11:42:56,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:42:56,900 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 11:42:56,900 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:42:56,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 11:42:57,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 11:42:57,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qhrh.c.gcp-upstream.internal','N') ;"
2026-09-24 11:42:57,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:42:57,805 [I] Table x_portal_user does not exist in database ranger
2026-09-24 11:42:57,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:42:58,039 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 11:42:58,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:42:58,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:42:58,514 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 11:42:58,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 11:42:59.358 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.372 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.386 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.398 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.413 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.429 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.476 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.482 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.493 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.502 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.510 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.522 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.530 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.536 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.542 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:42:59.547 UTC [12449] WARNING: there is no transaction in progress
2026-09-24 11:43:00,096 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 11:43:00,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 11:43:00,332 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 11:43:00,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:43:00,559 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 11:43:00,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:43:00,804 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 11:43:00,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:43:01,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:43:01,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qhrh.c.gcp-upstream.internal';"
2026-09-24 11:43:01,510 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 11:43:01,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 11:43:01,754 [I] DB_PATCHES have already been applied
I20260924 11:43:01.760412 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpreJ74/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 11:43:01.760496 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 11:43:01.774573 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:39659
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:43:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 11:43:02.886538 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.868s user 0.004s sys 0.032s
Sep 24, 2026 11:43:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:43:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 11:43:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:43:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:43:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 11:43:23.356173 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 11:43:23 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 11:43:23.486299 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 11:43:23.503276 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 11:43:24.266078 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 11:43:24.266179 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms
I20260924 11:43:24.266227 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:35741
2026-09-24 11:43:24,601 [I] DB FLAVOR :POSTGRES
2026-09-24 11:43:24,602 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:43:24,602 [I] Checking connection
2026-09-24 11:43:24,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 11:43:24,827 [I] connection success
2026-09-24 11:43:24,827 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 11:43:24,827 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 11:43:24,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 11:43:25,047 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 11:43:25,047 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:43:25,048 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 11:43:25,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35741/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 11:43:25,289 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 11:43:25.748436 388 mini_ranger_kms.cc:310] Created kms service
I20260924 11:43:25.901358 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 11:43:25.940060 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 11:43:25 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:26.289224 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 11:43:26.289531 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpreJ74/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpreJ74/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 11:43:26.289580 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 11:43:26.305876 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12629
I20260924 11:43:26.305994 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59067
Sep 24, 2026 11:43:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:43:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:43:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:43:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:43:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:43:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 11:43:28.467093 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.981s user 0.069s sys 0.274s
I20260924 11:43:28.467368 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 11:43:28.467561 388 mini_ranger_kms.cc:401] 127.0.97.20:59067/kms/v1/keys
Sep 24 11:43:28 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:43:28 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 11:43:29 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 11:43:31 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:43:31.419641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38987
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38987
--ranger_config_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:31.745242 12680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:31.745517 12680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:31.745658 12680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:31.752493 12680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:31.752583 12680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:31.752694 12680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:31.752789 12680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:31.752879 12680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:31.762398 12680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38987
--ranger_config_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:31.763545 12680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:31.764839 12680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:31.776335 12685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:31.776398 12688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:31.776665 12680 server_base.cc:1034] running on GCE node
W20260924 11:43:31.776423 12686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:31.777503 12680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:31.778736 12680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:31.779945 12680 hybrid_clock.cc:648] HybridClock initialized: now 1790250211779914 us; error 35 us; skew 500 ppm
I20260924 11:43:31.780283 12680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:31 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:31.783063 12680 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:31.784493 12680 webserver.cc:460] Webserver started at http://127.0.97.62:34203/ using document root <none> and password file <none>
I20260924 11:43:31.785054 12680 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:31.785176 12680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:31.785462 12680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:43:31 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:31.838913 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "1181b8a311c146049351016a4ef1d833"
format_stamp: "Formatted at 2026-09-24 11:43:31 on dist-test-slave-qhrh"
server_key: "698a9a49b59e40f295a09b4807fffb4c"
server_key_iv: "c203fdd2f64e148db0818697fc36de6a"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:31.839711 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "1181b8a311c146049351016a4ef1d833"
format_stamp: "Formatted at 2026-09-24 11:43:31 on dist-test-slave-qhrh"
server_key: "698a9a49b59e40f295a09b4807fffb4c"
server_key_iv: "c203fdd2f64e148db0818697fc36de6a"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:31.845216 12680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:43:31.858736 12695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:31.860090 12680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:43:31.860277 12680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "1181b8a311c146049351016a4ef1d833"
format_stamp: "Formatted at 2026-09-24 11:43:31 on dist-test-slave-qhrh"
server_key: "698a9a49b59e40f295a09b4807fffb4c"
server_key_iv: "c203fdd2f64e148db0818697fc36de6a"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:31.860479 12680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:31.878471 12680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:31.881975 12680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:31.882272 12680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:31.900450 12680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38987
I20260924 11:43:31.900468 12746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38987 every 8 connection(s)
I20260924 11:43:31.901780 12680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:31.905332 12747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:31.908216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 12680
I20260924 11:43:31.908425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:43:31.909011 388 external_mini_cluster.cc:1428] Setting key 43a0b0639fb46ad8bf8ab1622dd5d166
I20260924 11:43:31.916217 12747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Bootstrap starting.
I20260924 11:43:31.918955 12747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:31.919808 12747 log.cc:824] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:43:31 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:31.922222 12747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: No bootstrap required, opened a new log
I20260924 11:43:31.928272 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:31.910875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.0.97.62:38987)
0924 11:43:31.911551 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:31.911561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:31.911596 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:43:31.913144 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:31.913162 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:31.913506 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:31.913908 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:31.915116 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:31.916390 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:31.917665 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:31.918260 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:31.918437 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:31.921838 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:31.921845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:31.921860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:31.921894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:31.924764 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:31.925390 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:31.925395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:31.925401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:31.925540 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:31.925894 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:31.925901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:31.925909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:31.926365 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:31.926699 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:31.927039 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 11:43:31.927387 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":327,"threads_started":1}
I20260924 11:43:31.929646 12747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:31.930009 12747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:31.930133 12747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1181b8a311c146049351016a4ef1d833, State: Initialized, Role: FOLLOWER
I20260924 11:43:31.930696 12747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:31.930941 12747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:43:31.931037 12747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:43:31.931134 12747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:31.932966 12747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:31.933350 12747 leader_election.cc:302] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1181b8a311c146049351016a4ef1d833; no voters:
I20260924 11:43:31.933810 12747 leader_election.cc:288] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:43:31.933962 12752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:31.934612 12752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 LEADER]: Becoming Leader. State: Replica: 1181b8a311c146049351016a4ef1d833, State: Running, Role: LEADER
I20260924 11:43:31.935201 12752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:31.935739 12747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:31.938277 12753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1181b8a311c146049351016a4ef1d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } } }
I20260924 11:43:31.938762 12753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:31.938287 12754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1181b8a311c146049351016a4ef1d833. Latest consensus state: current_term: 1 leader_uuid: "1181b8a311c146049351016a4ef1d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } } }
I20260924 11:43:31.939280 12754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:31.941363 12761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:31.946162 12761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:31.946638 12747 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 11:43:31.964344 12761 catalog_manager.cc:1347] Generated new cluster ID: f91b893f0e0f4a21970dd0ab0b55c281
I20260924 11:43:31.964473 12761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:43:31.977625 12761 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:43:31.979228 12761 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:43:31.998122 12761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Generated new TSK 0
I20260924 11:43:31.998924 12761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:43:32.654892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38987
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:43:32 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 11:43:33.046571 12792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:33.046823 12792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:33.046969 12792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:33.056916 12792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:33.057027 12792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:33.057229 12792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:43:33.072203 12792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:33.073712 12792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:33.075325 12792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:33.090574 12802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:33.091898 12803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:33.100013 12792 server_base.cc:1034] running on GCE node
W20260924 11:43:33.100440 12805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:33.101213 12792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:33.102274 12792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:33.103551 12792 hybrid_clock.cc:648] HybridClock initialized: now 1790250213103483 us; error 55 us; skew 500 ppm
I20260924 11:43:33.103886 12792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:33 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:33.107890 12792 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:43:33.116310 12792 webserver.cc:460] Webserver started at http://127.0.97.1:43857/ using document root <none> and password file <none>
I20260924 11:43:33.117167 12792 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:33.117363 12792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:33.117739 12792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:43:33 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:33.149691 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "f40d23ec89e742e2a225102ad4447207"
format_stamp: "Formatted at 2026-09-24 11:43:33 on dist-test-slave-qhrh"
server_key: "45e4acb389f124cb9030457e20471a85"
server_key_iv: "f92a2350711d41878f7e846c747223ff"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:33.150482 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f40d23ec89e742e2a225102ad4447207"
format_stamp: "Formatted at 2026-09-24 11:43:33 on dist-test-slave-qhrh"
server_key: "45e4acb389f124cb9030457e20471a85"
server_key_iv: "f92a2350711d41878f7e846c747223ff"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:33.157823 12792 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260924 11:43:33.175282 12812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:33.177052 12792 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:43:33.177209 12792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "f40d23ec89e742e2a225102ad4447207"
format_stamp: "Formatted at 2026-09-24 11:43:33 on dist-test-slave-qhrh"
server_key: "45e4acb389f124cb9030457e20471a85"
server_key_iv: "f92a2350711d41878f7e846c747223ff"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:33.177475 12792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:33.241911 12792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:33.248694 12792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:33.249075 12792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:33.259840 12792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:33.272949 12792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:33.273093 12792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:33.273298 12792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:33.273430 12792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:43:33 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:33.733987 12924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42563 every 8 connection(s)
I20260924 11:43:33.734315 12792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42563
I20260924 11:43:33.736039 12792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:43:33.741588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 12792
I20260924 11:43:33.741787 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:43:33.742514 388 external_mini_cluster.cc:1428] Setting key 6fce8699a3db0ee1ba1a6f540a6d30af
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Sep 24 11:43:33 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 24 11:43:33 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:33.812264 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:33.754769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39123 (local address 127.0.97.62:38987)
0924 11:43:33.781025 (+ 26256us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:33.781031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:33.781049 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:43:33.781124 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:33.781129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:33.781209 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:33.781447 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:33.783338 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:33.784542 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:33.787355 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:33.787985 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:33.788106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:33.802397 (+ 14291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:33.802405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:33.802411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:33.802456 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:33.806878 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:33.807910 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:33.807915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:33.807918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:33.808057 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:33.810140 (+ 2083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:33.810146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:33.810149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:33.810483 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:33.810623 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:33.811644 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0924 11:43:33.811875 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26118,"thread_start_us":26548,"threads_started":1}
I20260924 11:43:33.813436 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:33.755641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38987 (local address 127.0.97.1:39123)
0924 11:43:33.766043 (+ 10402us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:33.766098 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:33.767866 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:33.781648 (+ 13782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:33.781662 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:33.782294 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:33.783117 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:33.783144 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:33.784712 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:33.784719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:33.787174 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:33.787189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:33.788347 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:33.788353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:33.788628 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:33.790130 (+ 1502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:33.790177 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:33.798839 (+ 8662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:33.807040 (+ 8201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:33.807061 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:33.807077 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:33.807670 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:33.809729 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:33.809733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:33.809736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:33.810010 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:33.810736 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:33.810744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:33.811162 (+ 418us) client_negotiation.cc:769] Sending connection context
0924 11:43:33.811496 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 11:43:33.812011 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10081,"thread_start_us":9823,"threads_started":1}
I20260924 11:43:33.815004 12925 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38987
I20260924 11:43:33.815284 12925 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:33.816051 12925 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:33.818331 12712 ts_manager.cc:194] Registered new tserver with Master: f40d23ec89e742e2a225102ad4447207 (127.0.97.1:42563)
I20260924 11:43:33.820219 12712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:43:33.843326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38987
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:34.232282 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:34.232506 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:34.232640 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:34.239897 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:34.239979 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:34.240130 12932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:43:34.248725 12932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:34.249748 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:34.250890 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:34.263271 12945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:34.263365 12948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:34.263293 12946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:34.263962 12932 server_base.cc:1034] running on GCE node
I20260924 11:43:34.264655 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:34.265466 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:34.266675 12932 hybrid_clock.cc:648] HybridClock initialized: now 1790250214266625 us; error 48 us; skew 500 ppm
I20260924 11:43:34.266952 12932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:34.270102 12932 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:43:34.271500 12932 webserver.cc:460] Webserver started at http://127.0.97.2:35645/ using document root <none> and password file <none>
I20260924 11:43:34.272086 12932 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:34.272236 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:34.272526 12932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:34.289454 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "ae0ab530c1c34b8ebe2253227a4fc056"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "e8f0d3747d524556b7e99221c40a5123"
server_key_iv: "bba742c48d7e738d9ad5e323cfa97d01"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:34.290524 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ae0ab530c1c34b8ebe2253227a4fc056"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "e8f0d3747d524556b7e99221c40a5123"
server_key_iv: "bba742c48d7e738d9ad5e323cfa97d01"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:34.297254 12932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 11:43:34.314088 12955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:34.315701 12932 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:43:34.315918 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "ae0ab530c1c34b8ebe2253227a4fc056"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "e8f0d3747d524556b7e99221c40a5123"
server_key_iv: "bba742c48d7e738d9ad5e323cfa97d01"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:34.316174 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:34.335899 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:34.339890 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:34.340250 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:34.341239 12932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:34.342734 12932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:34.342873 12932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:34.343025 12932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:34.343106 12932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:34.383487 12932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42213
I20260924 11:43:34.383507 13067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42213 every 8 connection(s)
I20260924 11:43:34.384814 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:43:34.388811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 12932
I20260924 11:43:34.389019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:43:34.389714 388 external_mini_cluster.cc:1428] Setting key c2daf95e57786f7c9dc3b80bee207b09
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:34.407503 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:34.387377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38053 (local address 127.0.97.62:38987)
0924 11:43:34.387910 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:34.387917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:34.391663 (+ 3746us) server_negotiation.cc:407] Connection header received
0924 11:43:34.392749 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:34.392754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:34.392877 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:34.392992 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:34.394311 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.394970 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.396440 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.396849 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.396970 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:34.399978 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:34.399984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:34.399987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:34.400024 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:34.403950 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:34.404860 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:34.404867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:34.404873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:34.405007 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:34.405537 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:34.405544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:34.405550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:34.405891 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:34.406029 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:34.406855 (+ 826us) server_negotiation.cc:299] Negotiation successful
0924 11:43:34.407076 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":229,"threads_started":1}
I20260924 11:43:34.408453 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:34.387900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38987 (local address 127.0.97.2:38053)
0924 11:43:34.390709 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:34.390752 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:34.392498 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:34.393226 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:34.393233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:34.393620 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:34.394151 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:34.394165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.395112 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.395117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:34.396306 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:34.396317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.397051 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.397058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:34.397269 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:34.397884 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:34.397903 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:34.399772 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:34.404117 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:34.404126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:34.404164 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:34.404727 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:34.405129 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:34.405135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:34.405139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:34.405403 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:34.406031 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:34.406039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:34.406406 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 11:43:34.406714 (+ 308us) client_negotiation.cc:240] Negotiation successful
0924 11:43:34.407169 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2575,"thread_start_us":2396,"threads_started":1}
I20260924 11:43:34.410040 13068 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38987
I20260924 11:43:34.410313 13068 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:34.411126 13068 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:34.412873 12712 ts_manager.cc:194] Registered new tserver with Master: ae0ab530c1c34b8ebe2253227a4fc056 (127.0.97.2:42213)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:43:34.414093 12712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:43:34.443658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38987
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:34.747780 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:34.748023 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:34.748188 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:34.754840 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:34.754957 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:34.755115 13074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:43:34.764309 13074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:34.765403 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:34.766499 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:34.778412 13079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:34.778407 13080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:34.778698 13082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:34.779111 13074 server_base.cc:1034] running on GCE node
I20260924 11:43:34.779597 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:34.780433 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:34.781658 13074 hybrid_clock.cc:648] HybridClock initialized: now 1790250214781599 us; error 49 us; skew 500 ppm
I20260924 11:43:34.781939 13074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:34.784849 13074 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:43:34.786048 13074 webserver.cc:460] Webserver started at http://127.0.97.3:36033/ using document root <none> and password file <none>
I20260924 11:43:34.786609 13074 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:34.786742 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:34.787020 13074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:34.801113 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "25bdc5b0f57a46f49fa9b20af5e262d3"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "b09183bff45baf46d7f995357ece0973"
server_key_iv: "ff69a71cf9d17ec82dcafa6f62c20e23"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:34.801946 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "25bdc5b0f57a46f49fa9b20af5e262d3"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "b09183bff45baf46d7f995357ece0973"
server_key_iv: "ff69a71cf9d17ec82dcafa6f62c20e23"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:34.806587 13074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:43:34.821359 13089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:34.822929 13074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:43:34.823083 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "25bdc5b0f57a46f49fa9b20af5e262d3"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "b09183bff45baf46d7f995357ece0973"
server_key_iv: "ff69a71cf9d17ec82dcafa6f62c20e23"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:34.823300 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:34.823494 12925 heartbeater.cc:503] Master 127.0.97.62:38987 was elected leader, sending a full tablet report...
I20260924 11:43:34.840057 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:34.843482 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:34.843770 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:34.844631 13074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:34.845731 13074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:34.845803 13074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:34.845942 13074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:34.845989 13074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:34.878461 13074 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46515
I20260924 11:43:34.878491 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46515 every 8 connection(s)
I20260924 11:43:34.879658 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:43:34.882627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13074
I20260924 11:43:34.882851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:43:34.883392 388 external_mini_cluster.cc:1428] Setting key 9abba995de71856cfdd3bf1f54e42359
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:34.898144 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:34.881965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50589 (local address 127.0.97.62:38987)
0924 11:43:34.882136 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:34.882140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:34.883513 (+ 1373us) server_negotiation.cc:407] Connection header received
0924 11:43:34.885259 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:34.885264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:34.885355 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:34.885426 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:34.886917 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.887574 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.889059 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.889520 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.889610 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:34.892626 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:34.892630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:34.892633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:34.892662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:34.895115 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:34.895804 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:34.895809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:34.895817 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:34.895939 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:34.896468 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:34.896476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:34.896479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:34.896736 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:34.896906 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:34.897583 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 11:43:34.897792 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:43:34.898905 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:34.882436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38987 (local address 127.0.97.3:50589)
0924 11:43:34.883296 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:34.883339 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:34.885039 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:34.885681 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:34.885693 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:34.886210 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:34.886776 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:34.886793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.887713 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.887717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:34.888934 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:34.888945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.889699 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.889704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:34.889875 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:34.890446 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:34.890468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:34.892421 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:34.895243 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:34.895250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:34.895262 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:34.895664 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:34.896053 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:34.896057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:34.896061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:34.896350 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:34.896884 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:34.896891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:34.897227 (+ 336us) client_negotiation.cc:769] Sending connection context
0924 11:43:34.897495 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 11:43:34.897924 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":421,"threads_started":1}
I20260924 11:43:34.900115 13202 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38987
I20260924 11:43:34.900411 13202 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:34.901023 13202 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:34.902401 12712 ts_manager.cc:194] Registered new tserver with Master: 25bdc5b0f57a46f49fa9b20af5e262d3 (127.0.97.3:46515)
I20260924 11:43:34.903545 12712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50589
I20260924 11:43:34.910463 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:43:34.912002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 12792
Sep 24 11:43:34 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:43:34.927459 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:34.912863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.97.1:42563)
0924 11:43:34.913411 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:34.913416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:34.913438 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:43:34.913488 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:34.913493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:34.913642 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:34.913760 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:34.915540 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.916620 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.917897 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:34.918374 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:34.918475 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:34.922255 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:34.922267 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:34.922278 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:34.922315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:34.925307 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:34.925861 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:34.925868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:34.925870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:34.926006 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:34.926315 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:34.926319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:34.926321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:34.926649 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:34.926872 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:34.927031 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 11:43:34.927203 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260924 11:43:34.928612 12920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:35.008849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 12792
I20260924 11:43:35.025854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 12932
Sep 24 11:43:35 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:43:35.040783 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:35.026706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.0.97.2:42213)
0924 11:43:35.027197 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:35.027203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:35.027225 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:43:35.027292 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:35.027299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:35.027490 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:35.027600 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:35.028912 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:35.029859 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:35.031103 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:35.031676 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:35.031778 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:35.035732 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:35.035742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:35.035755 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:35.035792 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:35.038542 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:35.039059 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:35.039065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:35.039067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:35.039189 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:35.039494 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:35.039497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:35.039499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:35.039786 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:35.039976 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:35.040226 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 11:43:35.040404 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
I20260924 11:43:35.042048 13063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:35.119174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 12932
I20260924 11:43:35.134256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13074
Sep 24 11:43:35 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:43:35.148895 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:35.135086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.97.3:46515)
0924 11:43:35.135587 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:35.135593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:35.135617 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:43:35.135722 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:35.135728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:35.135952 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:35.136109 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:35.137318 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:35.138280 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:35.139411 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:35.140035 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:35.140232 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:35.144298 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:35.144308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:35.144320 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:35.144352 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:35.146792 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:35.147290 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:35.147296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:35.147298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:35.147400 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:35.147687 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:35.147690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:35.147691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:35.147950 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:35.148294 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:35.148451 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 11:43:35.148632 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260924 11:43:35.150017 13197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:35.226629 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13074
I20260924 11:43:35.241628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 12680
I20260924 11:43:35.242826 12742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:35.342991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 12680
I20260924 11:43:35.363246 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38987
--webserver_interface=127.0.97.62
--webserver_port=34203
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38987
--ranger_config_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:35.658797 13213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:35.659029 13213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:35.659107 13213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:35.666374 13213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:35.666460 13213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:35.666507 13213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:35.666571 13213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:35.666652 13213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:35.675745 13213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38987
--ranger_config_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:35.676779 13213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:35.678045 13213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:35.688759 13218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:35.688840 13219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:35.688802 13221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:35.689662 13213 server_base.cc:1034] running on GCE node
I20260924 11:43:35.690348 13213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:35.691622 13213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:35.692849 13213 hybrid_clock.cc:648] HybridClock initialized: now 1790250215692780 us; error 55 us; skew 500 ppm
I20260924 11:43:35.693166 13213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:35 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:35.696069 13213 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:35.697374 13213 webserver.cc:460] Webserver started at http://127.0.97.62:34203/ using document root <none> and password file <none>
I20260924 11:43:35.697971 13213 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:35.698122 13213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:43:35 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:35.759076 13213 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:43:35.761849 13229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:35.763077 13213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:43:35.763259 13213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "1181b8a311c146049351016a4ef1d833"
format_stamp: "Formatted at 2026-09-24 11:43:31 on dist-test-slave-qhrh"
server_key: "698a9a49b59e40f295a09b4807fffb4c"
server_key_iv: "c203fdd2f64e148db0818697fc36de6a"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:35.764302 13213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:35.780905 13213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:35.784557 13213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:35.784858 13213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:35.801007 13213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38987
I20260924 11:43:35.801034 13280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38987 every 8 connection(s)
I20260924 11:43:35.802289 13213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:35.808274 13281 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:43:35.811053 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Bootstrap starting.
I20260924 11:43:35.812475 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13213
I20260924 11:43:35.813218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42563
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38987
--webserver_port=43857
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:43:35.828310 13281 log.cc:824] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:35.836004 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:43:35.836521 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Bootstrap complete.
I20260924 11:43:35.844743 13281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:35.845482 13281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1181b8a311c146049351016a4ef1d833, State: Initialized, Role: FOLLOWER
I20260924 11:43:35.846009 13281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:35.846251 13281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:43:35.846376 13281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:43:35.846530 13281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:43:35.849743 13281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:35.850178 13281 leader_election.cc:302] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1181b8a311c146049351016a4ef1d833; no voters:
I20260924 11:43:35.850735 13281 leader_election.cc:288] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:43:35.850857 13285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:43:35.852082 13285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [term 2 LEADER]: Becoming Leader. State: Replica: 1181b8a311c146049351016a4ef1d833, State: Running, Role: LEADER
I20260924 11:43:35.852588 13285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } }
I20260924 11:43:35.852790 13281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:35.855022 13286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1181b8a311c146049351016a4ef1d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } } }
I20260924 11:43:35.855398 13286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:35.856993 13290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:35.856813 13287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1181b8a311c146049351016a4ef1d833. Latest consensus state: current_term: 2 leader_uuid: "1181b8a311c146049351016a4ef1d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1181b8a311c146049351016a4ef1d833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38987 } } }
I20260924 11:43:35.857167 13287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:35.863116 13290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:35.864614 13290 catalog_manager.cc:1259] Loaded cluster ID: f91b893f0e0f4a21970dd0ab0b55c281
I20260924 11:43:35.864740 13290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:43:35.868150 13290 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:43:35.869381 13290 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Loaded TSK: 0
I20260924 11:43:35.870833 13290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:43:35.878057 13281 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 11:43:36.233407 13283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:36.233745 13283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:36.233889 13283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:36.242619 13283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:36.242767 13283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:36.242973 13283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:43:36.256506 13283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43857
--tserver_master_addrs=127.0.97.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:36.257668 13283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:36.258939 13283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:36.291469 13319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:36.293036 13321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:36.293207 13318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:36.294304 13283 server_base.cc:1034] running on GCE node
I20260924 11:43:36.295192 13283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:36.296397 13283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:36.300271 13283 hybrid_clock.cc:648] HybridClock initialized: now 1790250216300200 us; error 53 us; skew 500 ppm
I20260924 11:43:36.300676 13283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:36 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:36.305053 13283 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:43:36.308614 13283 webserver.cc:460] Webserver started at http://127.0.97.1:43857/ using document root <none> and password file <none>
I20260924 11:43:36.309424 13283 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:36.309578 13283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:43:36 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:36.351194 13283 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260924 11:43:36.364486 13329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:36.366187 13283 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260924 11:43:36.366406 13283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "f40d23ec89e742e2a225102ad4447207"
format_stamp: "Formatted at 2026-09-24 11:43:33 on dist-test-slave-qhrh"
server_key: "45e4acb389f124cb9030457e20471a85"
server_key_iv: "f92a2350711d41878f7e846c747223ff"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:36.367774 13283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:36.389084 13283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:36.394145 13283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:36.394503 13283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:36.401870 13283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:36.419059 13283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:36.419245 13283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:36.419450 13283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:36.419610 13283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:36.482183 13283 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42563
I20260924 11:43:36.482247 13442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42563 every 8 connection(s)
I20260924 11:43:36.484220 13283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:43:36.492527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13283
I20260924 11:43:36.493299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42213
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38987
--webserver_port=35645
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:43:36 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:36.529345 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:36.487590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38987 (local address 127.0.97.1:59495)
0924 11:43:36.500722 (+ 13132us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:36.500788 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:36.503950 (+ 3162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:36.507218 (+ 3268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:36.507230 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:36.507878 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:36.508737 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:36.508767 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:36.510430 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:36.510436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:36.513103 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:36.513122 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:36.515101 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:36.515110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:36.515443 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:36.516344 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:36.516381 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:36.519451 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:36.524704 (+ 5253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:36.524711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:36.524726 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:36.525270 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:36.525757 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:36.525762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:36.525765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:36.525986 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:36.526914 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:36.526921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:36.527288 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 11:43:36.527548 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 11:43:36.528046 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12845,"thread_start_us":12604,"threads_started":1}
I20260924 11:43:36.532194 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:36.488636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59495 (local address 127.0.97.62:38987)
0924 11:43:36.504665 (+ 16029us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:36.504684 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:36.504748 (+ 64us) server_negotiation.cc:407] Connection header received
0924 11:43:36.505378 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:36.505399 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:36.505926 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:36.506408 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:36.508972 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:36.510251 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:36.513317 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:36.514520 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:36.514708 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:36.519806 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:36.519817 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:36.519835 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:36.519884 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:36.524538 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:36.525427 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:36.525436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:36.525444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:36.525626 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:36.526104 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:36.526110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:36.526113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:36.526679 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:36.526849 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:36.530620 (+ 3771us) server_negotiation.cc:299] Negotiation successful
0924 11:43:36.530979 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15737,"thread_start_us":15903,"threads_started":1}
I20260924 11:43:36.535566 13443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38987
I20260924 11:43:36.535903 13443 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:36.536814 13443 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
W20260924 11:43:36.539674 13443 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38987 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:43:36.539917 13443 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:36.540241 13443 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:36.544482 13443 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:36.544809 13443 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
W20260924 11:43:36.545816 13443 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:43:36.864389 13447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:36.864703 13447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:36.864874 13447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:36.871836 13447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:36.871917 13447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:36.872063 13447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:43:36.882009 13447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35645
--tserver_master_addrs=127.0.97.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:36.883164 13447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:36.884418 13447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:36.898196 13460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:36.898878 13462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:36.899096 13459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:36.899329 13447 server_base.cc:1034] running on GCE node
I20260924 11:43:36.900000 13447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:36.901638 13447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:36.902887 13447 hybrid_clock.cc:648] HybridClock initialized: now 1790250216902838 us; error 36 us; skew 500 ppm
I20260924 11:43:36.903280 13447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:36 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:36.916324 13447 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:43:36.918164 13447 webserver.cc:460] Webserver started at http://127.0.97.2:35645/ using document root <none> and password file <none>
I20260924 11:43:36.919013 13447 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:36.919176 13447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:43:36 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:36.942122 13447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:43:36.945489 13472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:36.947098 13447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:43:36.947263 13447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "ae0ab530c1c34b8ebe2253227a4fc056"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "e8f0d3747d524556b7e99221c40a5123"
server_key_iv: "bba742c48d7e738d9ad5e323cfa97d01"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:36.948534 13447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:36.981487 13447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:36.986518 13447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:36.986838 13447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:36.988747 13447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:36.990228 13447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:36.990348 13447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:36.990480 13447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:36.990568 13447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:37.050179 13447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42213
I20260924 11:43:37.050251 13586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42213 every 8 connection(s)
I20260924 11:43:37.052049 13447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:43:37.062517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13447
I20260924 11:43:37.063314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46515
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38987
--webserver_port=36033
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:37.121387 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.055795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38987 (local address 127.0.97.2:60853)
0924 11:43:37.085810 (+ 30015us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:37.085863 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:37.100657 (+ 14794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:37.101963 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:37.101981 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:37.102730 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:37.103708 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.103740 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.105864 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.105869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.107702 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.107733 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.108960 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.108966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.109248 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.109943 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:37.109971 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:37.112769 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:37.116941 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:37.116949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:37.116967 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:37.117539 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:37.117943 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:37.117948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:37.117952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:37.118179 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:37.118879 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:37.118887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:37.119258 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 11:43:37.119529 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 11:43:37.120037 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29670,"thread_start_us":36833,"threads_started":1}
I20260924 11:43:37.122433 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.056425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60853 (local address 127.0.97.62:38987)
0924 11:43:37.085220 (+ 28795us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.085228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.089537 (+ 4309us) server_negotiation.cc:407] Connection header received
0924 11:43:37.101035 (+ 11498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.101053 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.101184 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.101277 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:37.104604 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.105671 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.107914 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.108568 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.108679 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.113048 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:37.113054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:37.113060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:37.113099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:37.116786 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.117670 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.117675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.117679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.117822 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.118365 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.118370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.118373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.118679 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:37.118807 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.121876 (+ 3069us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.122083 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28642,"thread_start_us":31712,"threads_started":1}
I20260924 11:43:37.124296 13587 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38987
I20260924 11:43:37.124598 13587 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:37.125372 13587 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:37.127302 13246 ts_manager.cc:194] Registered new tserver with Master: ae0ab530c1c34b8ebe2253227a4fc056 (127.0.97.2:42213)
I20260924 11:43:37.129418 13246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60853
W20260924 11:43:37.503651 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:37.503975 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:37.504227 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:37.514853 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:37.514966 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:37.515128 13591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:43:37.529176 13591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36033
--tserver_master_addrs=127.0.97.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:37.530682 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:37.532234 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:43:37.550369 13443 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:37.550827 13443 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:37.552016 13246 ts_manager.cc:194] Registered new tserver with Master: f40d23ec89e742e2a225102ad4447207 (127.0.97.1:42563)
I20260924 11:43:37.552960 13246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59495
W20260924 11:43:37.564541 13598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:37.565047 13600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:37.565405 13597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:37.566396 13591 server_base.cc:1034] running on GCE node
I20260924 11:43:37.567058 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:37.568164 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:37.569430 13591 hybrid_clock.cc:648] HybridClock initialized: now 1790250217569387 us; error 39 us; skew 500 ppm
I20260924 11:43:37.569809 13591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:37.573216 13591 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:43:37.577447 13591 webserver.cc:460] Webserver started at http://127.0.97.3:36033/ using document root <none> and password file <none>
I20260924 11:43:37.586699 13591 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:37.586846 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:37.643179 13591 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:43:37.645980 13608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:37.647261 13591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:43:37.647426 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "25bdc5b0f57a46f49fa9b20af5e262d3"
format_stamp: "Formatted at 2026-09-24 11:43:34 on dist-test-slave-qhrh"
server_key: "b09183bff45baf46d7f995357ece0973"
server_key_iv: "ff69a71cf9d17ec82dcafa6f62c20e23"
server_key_version: "kuduclusterkey@0"
I20260924 11:43:37.648303 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:37.671206 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:37.675204 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:37.675457 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:37.676440 13591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:37.677798 13591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:37.677874 13591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:37.678006 13591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:37.678082 13591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:37.721591 13591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46515
I20260924 11:43:37.721670 13720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46515 every 8 connection(s)
I20260924 11:43:37.723479 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:43:37.729121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13591
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:37.769349 13724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.727742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41311 (local address 127.0.97.62:38987)
0924 11:43:37.746222 (+ 18480us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.746230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.747955 (+ 1725us) server_negotiation.cc:407] Connection header received
0924 11:43:37.750347 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.750352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.750430 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.750548 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:37.752414 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.753625 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.755884 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.756429 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.756530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.762066 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:37.762072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:37.762077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:37.762112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:37.765926 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.766666 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.766674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.766676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.767039 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.767494 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.767498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.767501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.767797 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:37.767922 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.768779 (+ 857us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.768982 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18326,"thread_start_us":18476,"threads_started":1}
I20260924 11:43:37.770147 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.727047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38987 (local address 127.0.97.3:41311)
0924 11:43:37.747524 (+ 20477us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:37.747570 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:37.750042 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:37.750966 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:37.750976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:37.751461 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:37.752181 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.752203 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.753799 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.753803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.755736 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.755757 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.756802 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.756808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.757051 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.758316 (+ 1265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:37.759089 (+ 773us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:37.761807 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:37.766061 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:37.766069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:37.766082 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:37.766528 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:37.767169 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:37.767173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:37.767176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:37.767378 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:37.767995 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:37.768001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:37.768327 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 11:43:37.768595 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 11:43:37.769031 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20235,"thread_start_us":20002,"threads_started":1}
I20260924 11:43:37.771612 13721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38987
I20260924 11:43:37.771896 13721 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:37.772670 13721 heartbeater.cc:511] Master 127.0.97.62:38987 requested a full tablet report, sending...
I20260924 11:43:37.774302 13246 ts_manager.cc:194] Registered new tserver with Master: 25bdc5b0f57a46f49fa9b20af5e262d3 (127.0.97.3:46515)
I20260924 11:43:37.775302 13246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41311
I20260924 11:43:37.779008 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.746398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.97.62:38987)
0924 11:43:37.748401 (+ 2003us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.748407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.748422 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:43:37.748497 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.748501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.748597 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.748692 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:37.752185 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.753682 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.757530 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.758167 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.758276 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.763865 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:37.763871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:37.763876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:37.763922 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:37.768340 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.771667 (+ 3327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.771673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.771677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.771860 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.776086 (+ 4226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.776091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.776094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.776419 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:37.776666 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.778308 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.778495 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":1883,"thread_start_us":2392,"threads_started":1}
I20260924 11:43:37.781471 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:43:37.807972 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.792194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.97.62:38987)
0924 11:43:37.792336 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.792342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.793046 (+ 704us) server_negotiation.cc:407] Connection header received
0924 11:43:37.793176 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.793181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.793260 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.793359 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:37.795177 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.796061 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.797765 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.798329 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.798448 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.801546 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:37.801552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:37.801557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:37.801590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:37.805090 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.805863 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.805868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.805872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.805987 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:37.806376 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:37.806381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:37.806384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:37.806692 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:37.806902 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.807445 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.807647 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 11:43:37.817241 13246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:43:37.828292 13246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 24 11:43:37 dist-test-slave-qhrh krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 11:43:37.865734 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.852374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44232)
0924 11:43:37.853624 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:37.853660 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:37.853831 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:37.855564 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:37.855572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:37.855610 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:37.856026 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.856043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.859077 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.859085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.861958 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.861972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.863831 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.863837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.864031 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.864081 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 11:43:37.864244 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 11:43:37.864500 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":888,"threads_started":1}
I20260924 11:43:37.868211 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.853108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.0.97.1:42563)
0924 11:43:37.854735 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.854743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.854770 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:43:37.854856 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.854864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.855324 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.856280 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:37.856561 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.858902 (+ 2341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.862170 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.863599 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.866742 (+ 3143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.867039 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.867145 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.867703 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":980,"threads_started":1}
I20260924 11:43:37.872102 13378 tablet_service.cc:1467] Processing CreateTablet for tablet c18157edf1214e249050dcdd465a0138 (DEFAULT_TABLE table=test-table [id=f426046bb91a4feb99e7b64e89cb9425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:37.873607 13378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18157edf1214e249050dcdd465a0138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:37.887632 13742 tablet_bootstrap.cc:492] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: Bootstrap starting.
I20260924 11:43:37.890820 13742 tablet_bootstrap.cc:654] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:37.893504 13742 log.cc:824] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:37.902530 13742 tablet_bootstrap.cc:492] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: No bootstrap required, opened a new log
I20260924 11:43:37.902871 13742 ts_tablet_manager.cc:1397] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260924 11:43:37.910669 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.855206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46515 (local address 127.0.0.1:41078)
0924 11:43:37.866394 (+ 11188us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:37.866415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:37.866540 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:37.870287 (+ 3747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:37.870292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:37.870321 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:37.870467 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.870476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.872527 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.872532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.874294 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.874307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.909965 (+ 35658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.909972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.910156 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.910196 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:43:37.910260 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 11:43:37.910403 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11059,"thread_start_us":12223,"threads_started":1}
I20260924 11:43:37.911996 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.855814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.97.3:46515)
0924 11:43:37.869729 (+ 13915us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.869736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.869764 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:43:37.869850 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.869857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.870066 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.870619 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:37.870871 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.872397 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.874876 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.908723 (+ 33847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.911227 (+ 2504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.911444 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.911520 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.911685 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13767,"thread_start_us":31173,"threads_started":1}
I20260924 11:43:37.913323 13742 raft_consensus.cc:348] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:37.913834 13742 raft_consensus.cc:374] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:37.913975 13742 raft_consensus.cc:729] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40d23ec89e742e2a225102ad4447207, State: Initialized, Role: FOLLOWER
I20260924 11:43:37.914671 13742 consensus_queue.cc:260] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:37.915680 13656 tablet_service.cc:1467] Processing CreateTablet for tablet c18157edf1214e249050dcdd465a0138 (DEFAULT_TABLE table=test-table [id=f426046bb91a4feb99e7b64e89cb9425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:37.917135 13443 heartbeater.cc:503] Master 127.0.97.62:38987 was elected leader, sending a full tablet report...
I20260924 11:43:37.917651 13742 ts_tablet_manager.cc:1428] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
I20260924 11:43:37.919523 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.874298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42213 (local address 127.0.0.1:56266)
0924 11:43:37.874590 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:37.874606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:37.874702 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:37.912636 (+ 37934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:37.912640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:37.912665 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:37.912959 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.912970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.914898 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.914915 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.916809 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:37.916822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.918792 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.918799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:37.918958 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.918992 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:43:37.919081 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 11:43:37.919215 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 11:43:37.920267 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:37.868046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.97.2:42213)
0924 11:43:37.912068 (+ 44022us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:37.912078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:37.912110 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:43:37.912215 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:37.912222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:37.912436 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:37.912649 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:37.913179 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.914721 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.916988 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:37.918343 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:37.918459 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:37.918636 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:37.919779 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0924 11:43:37.919936 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43877,"thread_start_us":43770,"threads_started":1}
I20260924 11:43:37.923856 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18157edf1214e249050dcdd465a0138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:37.924374 13522 tablet_service.cc:1467] Processing CreateTablet for tablet c18157edf1214e249050dcdd465a0138 (DEFAULT_TABLE table=test-table [id=f426046bb91a4feb99e7b64e89cb9425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:37.925562 13522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18157edf1214e249050dcdd465a0138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:37.944753 13745 tablet_bootstrap.cc:492] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Bootstrap starting.
I20260924 11:43:37.949244 13745 tablet_bootstrap.cc:654] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:37.949957 13746 tablet_bootstrap.cc:492] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Bootstrap starting.
I20260924 11:43:37.954387 13745 log.cc:824] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:37.955423 13746 tablet_bootstrap.cc:654] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:37.956580 13746 log.cc:824] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:37.965116 13746 tablet_bootstrap.cc:492] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: No bootstrap required, opened a new log
I20260924 11:43:37.965400 13746 ts_tablet_manager.cc:1397] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260924 11:43:37.966979 13745 tablet_bootstrap.cc:492] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: No bootstrap required, opened a new log
I20260924 11:43:37.967263 13745 ts_tablet_manager.cc:1397] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.005s
I20260924 11:43:37.975597 13746 raft_consensus.cc:348] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:37.976145 13746 raft_consensus.cc:374] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:37.976336 13746 raft_consensus.cc:729] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0ab530c1c34b8ebe2253227a4fc056, State: Initialized, Role: FOLLOWER
I20260924 11:43:37.977061 13746 consensus_queue.cc:260] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:37.980868 13745 raft_consensus.cc:348] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:37.981297 13745 raft_consensus.cc:374] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:37.981477 13745 raft_consensus.cc:729] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bdc5b0f57a46f49fa9b20af5e262d3, State: Initialized, Role: FOLLOWER
I20260924 11:43:37.982214 13745 consensus_queue.cc:260] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:37.988857 13587 heartbeater.cc:503] Master 127.0.97.62:38987 was elected leader, sending a full tablet report...
I20260924 11:43:37.989356 13746 ts_tablet_manager.cc:1428] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
W20260924 11:43:37.993109 13444 tablet.cc:2369] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:37.993228 13721 heartbeater.cc:503] Master 127.0.97.62:38987 was elected leader, sending a full tablet report...
I20260924 11:43:37.994789 13745 ts_tablet_manager.cc:1428] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Time spent starting tablet: real 0.027s user 0.009s sys 0.012s
W20260924 11:43:37.999634 13722 tablet.cc:2369] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:38.023366 13749 raft_consensus.cc:482] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:43:38.023654 13749 raft_consensus.cc:504] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:38.025591 13749 leader_election.cc:288] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40d23ec89e742e2a225102ad4447207 (127.0.97.1:42563), 25bdc5b0f57a46f49fa9b20af5e262d3 (127.0.97.3:46515)
I20260924 11:43:38.076025 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.026585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40129 (local address 127.0.97.1:42563)
0924 11:43:38.026782 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:38.026788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:38.051958 (+ 25170us) server_negotiation.cc:407] Connection header received
0924 11:43:38.061626 (+ 9668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:38.061633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:38.061727 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:38.061793 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:38.066044 (+ 4251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.066715 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.071787 (+ 5072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.072957 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.073080 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.073181 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:38.075655 (+ 2474us) server_negotiation.cc:299] Negotiation successful
0924 11:43:38.075781 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:43:38.076886 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.026137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.97.2:40129)
0924 11:43:38.051845 (+ 25708us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.051866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:38.060576 (+ 8710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:38.061982 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:38.061986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:38.062007 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:38.062341 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:38.062352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.066917 (+ 4565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.066920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:38.071485 (+ 4565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:38.071498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.073427 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.073433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:38.073602 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.073646 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:43:38.076463 (+ 2817us) client_negotiation.cc:240] Negotiation successful
0924 11:43:38.076587 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25568,"thread_start_us":25962,"threads_started":1}
I20260924 11:43:38.077598 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.027378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48627 (local address 127.0.97.3:46515)
0924 11:43:38.027575 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:38.027581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:38.060299 (+ 32718us) server_negotiation.cc:407] Connection header received
0924 11:43:38.060750 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:38.060755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:38.060828 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:38.060891 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:38.062501 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.063710 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.073791 (+ 10081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.074947 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.075036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.075127 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:38.077162 (+ 2035us) server_negotiation.cc:299] Negotiation successful
0924 11:43:38.077355 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:43:38.078501 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.026966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46515 (local address 127.0.97.2:48627)
0924 11:43:38.059008 (+ 32042us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.060205 (+ 1197us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:38.060411 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:38.061080 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:38.061085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:38.061108 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:38.061419 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:38.061429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.063887 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.063892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:38.065859 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:38.065884 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.075302 (+ 9418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.075308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:38.075463 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.075493 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:43:38.078080 (+ 2587us) client_negotiation.cc:240] Negotiation successful
0924 11:43:38.078217 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31915,"thread_start_us":44604,"threads_started":1}
I20260924 11:43:38.081959 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18157edf1214e249050dcdd465a0138" candidate_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40d23ec89e742e2a225102ad4447207" is_pre_election: true
I20260924 11:43:38.082585 13398 raft_consensus.cc:2454] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0ab530c1c34b8ebe2253227a4fc056 in term 0.
I20260924 11:43:38.083261 13474 leader_election.cc:302] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0ab530c1c34b8ebe2253227a4fc056, f40d23ec89e742e2a225102ad4447207; no voters:
I20260924 11:43:38.083411 13676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18157edf1214e249050dcdd465a0138" candidate_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" is_pre_election: true
I20260924 11:43:38.083748 13749 raft_consensus.cc:2781] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:43:38.083875 13749 raft_consensus.cc:482] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:43:38.083979 13749 raft_consensus.cc:3037] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:38.083904 13676 raft_consensus.cc:2454] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0ab530c1c34b8ebe2253227a4fc056 in term 0.
I20260924 11:43:38.086484 13749 raft_consensus.cc:504] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:38.087002 13749 leader_election.cc:288] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [CANDIDATE]: Term 1 election: Requested vote from peers f40d23ec89e742e2a225102ad4447207 (127.0.97.1:42563), 25bdc5b0f57a46f49fa9b20af5e262d3 (127.0.97.3:46515)
I20260924 11:43:38.087607 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18157edf1214e249050dcdd465a0138" candidate_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40d23ec89e742e2a225102ad4447207"
I20260924 11:43:38.087785 13398 raft_consensus.cc:3037] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:38.087874 13676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18157edf1214e249050dcdd465a0138" candidate_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3"
I20260924 11:43:38.088032 13676 raft_consensus.cc:3037] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:38.090700 13398 raft_consensus.cc:2454] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0ab530c1c34b8ebe2253227a4fc056 in term 1.
I20260924 11:43:38.090799 13676 raft_consensus.cc:2454] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0ab530c1c34b8ebe2253227a4fc056 in term 1.
I20260924 11:43:38.091475 13476 leader_election.cc:302] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bdc5b0f57a46f49fa9b20af5e262d3, ae0ab530c1c34b8ebe2253227a4fc056; no voters:
I20260924 11:43:38.092048 13749 raft_consensus.cc:2781] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:38.092617 13749 raft_consensus.cc:686] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 1 LEADER]: Becoming Leader. State: Replica: ae0ab530c1c34b8ebe2253227a4fc056, State: Running, Role: LEADER
I20260924 11:43:38.093209 13749 consensus_queue.cc:237] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } }
I20260924 11:43:38.097796 13245 catalog_manager.cc:5495] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0ab530c1c34b8ebe2253227a4fc056 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae0ab530c1c34b8ebe2253227a4fc056" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42213 } health_report { overall_health: HEALTHY } } }
W20260924 11:43:38.105345 13588 tablet.cc:2369] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:38.130987 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.125503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.0.97.2:42213)
0924 11:43:38.125684 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:38.125691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:38.125899 (+ 208us) server_negotiation.cc:407] Connection header received
0924 11:43:38.126125 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:38.126131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:38.126287 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:38.126447 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:43:38.126802 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.127496 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.128985 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.129369 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.129486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.129979 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:43:38.130434 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:43:38.130579 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:38.130635 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 11:43:38.130718 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:43:38.139940 13676 raft_consensus.cc:1261] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 1 FOLLOWER]: Refusing update from remote peer ae0ab530c1c34b8ebe2253227a4fc056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:38.140143 13398 raft_consensus.cc:1261] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 1 FOLLOWER]: Refusing update from remote peer ae0ab530c1c34b8ebe2253227a4fc056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:38.140913 13753 consensus_queue.cc:1035] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bdc5b0f57a46f49fa9b20af5e262d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:38.141260 13749 consensus_queue.cc:1035] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40d23ec89e742e2a225102ad4447207" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:38.154080 13760 mvcc.cc:205] Tried to move back new op lower bound from 7332864893491187712 to 7332864893314109440. Current Snapshot: MvccSnapshot[applied={T|T < 7332864893491187712}]
I20260924 11:43:38.177435 13245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:43:38.177736 13245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:43:38.181849 13245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1181b8a311c146049351016a4ef1d833: Sending DeleteTablet for 3 replicas of tablet c18157edf1214e249050dcdd465a0138
I20260924 11:43:38.182884 13378 tablet_service.cc:1514] Processing DeleteTablet for tablet c18157edf1214e249050dcdd465a0138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44232
I20260924 11:43:38.183503 13656 tablet_service.cc:1514] Processing DeleteTablet for tablet c18157edf1214e249050dcdd465a0138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41078
I20260924 11:43:38.183827 13522 tablet_service.cc:1514] Processing DeleteTablet for tablet c18157edf1214e249050dcdd465a0138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56266
I20260924 11:43:38.190676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13283
I20260924 11:43:38.229780 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.191674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.0.97.1:42563)
0924 11:43:38.191833 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:38.191838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:38.191912 (+ 74us) server_negotiation.cc:407] Connection header received
0924 11:43:38.192110 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:38.192117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:38.192227 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:38.192327 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:38.194611 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.195718 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.208872 (+ 13154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.209654 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.209798 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.216823 (+ 7025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:38.216840 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:38.216862 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:38.216912 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:38.221323 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:38.224751 (+ 3428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:38.224763 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:38.224769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:38.224971 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:38.227165 (+ 2194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:38.227174 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:38.227179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:38.227721 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:38.228041 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:38.229094 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0924 11:43:38.229317 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:43:38.230659 13438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:38.231941 13774 tablet_replica.cc:330] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: stopping tablet replica
I20260924 11:43:38.232650 13774 raft_consensus.cc:2229] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:43:38.233645 13774 raft_consensus.cc:2258] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:38.235733 13773 tablet_replica.cc:330] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: stopping tablet replica
I20260924 11:43:38.237039 13773 raft_consensus.cc:2229] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:43:38.238421 13773 raft_consensus.cc:2258] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:38.241782 13774 ts_tablet_manager.cc:1905] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:38.243623 13773 ts_tablet_manager.cc:1905] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:38.252899 13774 ts_tablet_manager.cc:1918] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:38.253145 13774 log.cc:1196] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/c18157edf1214e249050dcdd465a0138
I20260924 11:43:38.253685 13774 ts_tablet_manager.cc:1939] T c18157edf1214e249050dcdd465a0138 P ae0ab530c1c34b8ebe2253227a4fc056: Deleting consensus metadata
I20260924 11:43:38.253883 13773 ts_tablet_manager.cc:1918] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:38.254148 13773 log.cc:1196] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/c18157edf1214e249050dcdd465a0138
I20260924 11:43:38.254700 13773 ts_tablet_manager.cc:1939] T c18157edf1214e249050dcdd465a0138 P 25bdc5b0f57a46f49fa9b20af5e262d3: Deleting consensus metadata
I20260924 11:43:38.255540 13233 catalog_manager.cc:4841] TS ae0ab530c1c34b8ebe2253227a4fc056 (127.0.97.2:42213): tablet c18157edf1214e249050dcdd465a0138 (table test-table [id=f426046bb91a4feb99e7b64e89cb9425]) successfully deleted
I20260924 11:43:38.257203 13233 catalog_manager.cc:4841] TS 25bdc5b0f57a46f49fa9b20af5e262d3 (127.0.97.3:46515): tablet c18157edf1214e249050dcdd465a0138 (table test-table [id=f426046bb91a4feb99e7b64e89cb9425]) successfully deleted
I20260924 11:43:38.341850 13772 tablet_replica.cc:330] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207: stopping tablet replica
I20260924 11:43:38.342336 13772 raft_consensus.cc:2229] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:43:38.342862 13772 raft_consensus.cc:2258] T c18157edf1214e249050dcdd465a0138 P f40d23ec89e742e2a225102ad4447207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:38.343664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13283
W20260924 11:43:38.361404 13231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42563 (error 108)
I20260924 11:43:38.363209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13447
I20260924 11:43:38.363240 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.362714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44248)
0924 11:43:38.362919 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.363054 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 11:43:38.363754 13231 catalog_manager.cc:4584] TS f40d23ec89e742e2a225102ad4447207 (127.0.97.1:42563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c18157edf1214e249050dcdd465a0138: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
I20260924 11:43:38.377775 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.364554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.0.97.2:42213)
0924 11:43:38.364725 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:38.364729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:38.364744 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:43:38.364806 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:38.364810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:38.364881 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:38.365017 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:38.366364 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.367279 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.368927 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.369412 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.369543 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.372038 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:38.372049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:38.372062 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:38.372092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:38.375062 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:38.375695 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:38.375707 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:38.375709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:38.375833 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:38.376195 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:38.376202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:38.376207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:38.376593 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:38.376917 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:38.377227 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 11:43:38.377414 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:43:38.378574 13582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:38.417495 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.416977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44254)
0924 11:43:38.417210 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.417316 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 11:43:38.418603 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.418233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44268)
0924 11:43:38.418390 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.418471 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:43:38.479382 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.478827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44272)
0924 11:43:38.479098 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.479197 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 11:43:38.480623 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.480211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44282)
0924 11:43:38.480386 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.480486 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 11:43:38.489527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13447
I20260924 11:43:38.513630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13591
I20260924 11:43:38.526269 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.514412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.0.97.3:46515)
0924 11:43:38.514584 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:38.514588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:38.514606 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:43:38.514816 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:38.514820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:38.514876 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:38.514950 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:38.516502 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.517144 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.518360 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:38.518683 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:38.518780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:38.521024 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:38.521036 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:38.521051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:38.521082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:38.523666 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:38.524363 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:38.524372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:38.524376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:38.524505 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:38.524856 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:38.524861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:38.524864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:38.525190 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:38.525478 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:38.525766 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 11:43:38.525961 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:43:38.526865 13716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:38.561022 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.560592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44292)
0924 11:43:38.560783 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.560865 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 11:43:38.561995 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:38.561625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42563 (local address 127.0.0.1:44308)
0924 11:43:38.561826 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:38.561888 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 11:43:38.609974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13591
I20260924 11:43:38.625345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13213
I20260924 11:43:38.626415 13276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:38.728413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13213
I20260924 11:43:38.748291 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 11:43:39.083914 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 11:43:39.084199 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 11:43:39.431005 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 11:43:39.431 UTC [12266] LOG: received smart shutdown request
2026-09-24 11:43:39.435 UTC [12266] LOG: background worker "logical replication launcher" (PID 12275) exited with exit code 1
2026-09-24 11:43:39.435 UTC [12270] LOG: shutting down
2026-09-24 11:43:39.580 UTC [12266] LOG: database system is shut down
2026-09-24T11:43:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49254 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:43:39 dist-test-slave-qhrh krb5kdc[13788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:43:39 dist-test-slave-qhrh krb5kdc[13788](info): set up 2 sockets
Sep 24 11:43:39 dist-test-slave-qhrh krb5kdc[13788](info): commencing operation
krb5kdc: starting...
W20260924 11:43:40.720288 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:43:40 dist-test-slave-qhrh krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:43:40.905401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45393
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45393,127.0.97.61:38935,127.0.97.60:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:41.215760 13801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:41.216044 13801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:41.216180 13801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:41.222931 13801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:41.223057 13801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:41.223150 13801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:41.223282 13801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:41.223371 13801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:41.232251 13801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45393,127.0.97.61:38935,127.0.97.60:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:41.233379 13801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:41.234623 13801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:41.245779 13807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:41.245811 13809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:41.245844 13806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:41.246948 13801 server_base.cc:1034] running on GCE node
I20260924 11:43:41.247599 13801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:41.248821 13801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:41.250044 13801 hybrid_clock.cc:648] HybridClock initialized: now 1790250221250019 us; error 42 us; skew 500 ppm
I20260924 11:43:41.250340 13801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:41 dist-test-slave-qhrh krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:41.253326 13801 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:41.254643 13801 webserver.cc:460] Webserver started at http://127.0.97.62:33981/ using document root <none> and password file <none>
I20260924 11:43:41.255277 13801 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:41.255415 13801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:41.255697 13801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:41.258353 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "0694a96692f64996881b89e1892da9f3"
format_stamp: "Formatted at 2026-09-24 11:43:41 on dist-test-slave-qhrh"
server_key: "07488181d7c78639a8591217ad4ce493"
server_key_iv: "4a867a2e4bdede7e5b0f56c2c789bc6e"
server_key_version: "encryptionkey@0"
I20260924 11:43:41.259056 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "0694a96692f64996881b89e1892da9f3"
format_stamp: "Formatted at 2026-09-24 11:43:41 on dist-test-slave-qhrh"
server_key: "07488181d7c78639a8591217ad4ce493"
server_key_iv: "4a867a2e4bdede7e5b0f56c2c789bc6e"
server_key_version: "encryptionkey@0"
I20260924 11:43:41.263636 13801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260924 11:43:41.266759 13816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:41.267855 13801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:43:41.268024 13801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "0694a96692f64996881b89e1892da9f3"
format_stamp: "Formatted at 2026-09-24 11:43:41 on dist-test-slave-qhrh"
server_key: "07488181d7c78639a8591217ad4ce493"
server_key_iv: "4a867a2e4bdede7e5b0f56c2c789bc6e"
server_key_version: "encryptionkey@0"
I20260924 11:43:41.268244 13801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:41.286710 13801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:41.290233 13801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:41.290485 13801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:41.308238 13801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45393
I20260924 11:43:41.308349 13867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45393 every 8 connection(s)
I20260924 11:43:41.309423 13801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:41.311185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13801
I20260924 11:43:41.311374 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:43:41.312005 388 external_mini_cluster.cc:1428] Setting key 2d62ababfdedac138273383d8766ceb9
I20260924 11:43:41.312974 13868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:41.319020 13868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 11:43:41 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:41.341682 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.322763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.0.97.62:45393)
0924 11:43:41.323429 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:41.323439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:41.323485 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:43:41.324717 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:41.324727 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:41.325000 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:41.325199 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:41.326171 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.328268 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.329572 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.330439 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.330642 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:41.334457 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:41.334468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:41.334484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:41.334522 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:41.338194 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:41.338805 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:41.338815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:41.338822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:41.339003 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:41.339376 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:41.339383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:41.339387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:41.339804 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:41.339996 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:41.340543 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 11:43:41.340754 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
I20260924 11:43:41.341710 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.321644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45393 (local address 127.0.0.1:49762)
0924 11:43:41.322387 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.322428 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:41.324375 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:41.325223 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:41.325227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:41.325699 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:41.325992 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:41.326006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.328444 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.328448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:41.329436 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:41.329446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.330665 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.330675 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:41.330833 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:41.331364 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:41.331383 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:41.334207 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:41.338310 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:41.338319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:41.338326 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:41.338682 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:41.339133 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:41.339136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:41.339138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:41.339267 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:41.339931 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:41.339938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:41.340191 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 11:43:41.340444 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 11:43:41.340910 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":45,"thread_start_us":318,"threads_started":1}
I20260924 11:43:41.344616 13868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:43:41.345574 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.345255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55808)
0924 11:43:41.345385 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.345469 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:43:41.345911 13819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
I20260924 11:43:41.347291 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.347009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55822)
0924 11:43:41.347125 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.347188 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 11:43:41.347529 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38935: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 11:43:41.358501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38935
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45393,127.0.97.61:38935,127.0.97.60:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:43:41.396835 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } attempt: 1
I20260924 11:43:41.397889 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.397496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55836)
0924 11:43:41.397665 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.397761 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:43:41.398864 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.398504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55848)
0924 11:43:41.398641 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.398711 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:43:41.399037 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38935: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
I20260924 11:43:41.467280 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } attempt: 2
I20260924 11:43:41.468426 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.467901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55858)
0924 11:43:41.468199 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.468288 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:43:41.469344 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.469020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55860)
0924 11:43:41.469155 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.469231 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:43:41.469515 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38935: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
I20260924 11:43:41.560748 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } attempt: 3
I20260924 11:43:41.562353 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.561747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55870)
0924 11:43:41.562052 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.562173 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260924 11:43:41.563349 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.562980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55884)
0924 11:43:41.563130 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.563206 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:43:41.563617 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38935: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
W20260924 11:43:41.672338 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:41.672581 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:41.672719 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:41.679265 13875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:41.679353 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:41.679394 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:41.679461 13875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:41.679527 13875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:41.688383 13875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45393,127.0.97.61:38935,127.0.97.60:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:41.689440 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:41.690621 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:41.700977 13880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:41.701052 13883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:41.701107 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:41.701670 13875 server_base.cc:1034] running on GCE node
I20260924 11:43:41.702360 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:41.703545 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:41.704759 13875 hybrid_clock.cc:648] HybridClock initialized: now 1790250221704726 us; error 48 us; skew 500 ppm
I20260924 11:43:41.705047 13875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:43:41.706869 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } attempt: 4
Sep 24 11:43:41 dist-test-slave-qhrh krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:41.707921 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.707412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55892)
0924 11:43:41.707722 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.707800 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 11:43:41.708027 13875 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 11:43:41.709034 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.708690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55906)
0924 11:43:41.708835 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.708924 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:43:41.709409 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38935: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38935: connect: Connection refused (error 111)
I20260924 11:43:41.709611 13875 webserver.cc:460] Webserver started at http://127.0.97.61:40755/ using document root <none> and password file <none>
I20260924 11:43:41.710180 13875 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:41.710301 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:41.710568 13875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:41.713142 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data/instance:
uuid: "0f183e831dff48c39464cb48e953d40b"
format_stamp: "Formatted at 2026-09-24 11:43:41 on dist-test-slave-qhrh"
server_key: "7ef5a3841c98ff035dcf671060191538"
server_key_iv: "4876201f7e4da768248d1fa1cd8c9cda"
server_key_version: "encryptionkey@0"
I20260924 11:43:41.713871 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal/instance:
uuid: "0f183e831dff48c39464cb48e953d40b"
format_stamp: "Formatted at 2026-09-24 11:43:41 on dist-test-slave-qhrh"
server_key: "7ef5a3841c98ff035dcf671060191538"
server_key_iv: "4876201f7e4da768248d1fa1cd8c9cda"
server_key_version: "encryptionkey@0"
I20260924 11:43:41.718494 13875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 11:43:41.721577 13890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:41.722703 13875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:43:41.722872 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal
uuid: "0f183e831dff48c39464cb48e953d40b"
format_stamp: "Formatted at 2026-09-24 11:43:41 on dist-test-slave-qhrh"
server_key: "7ef5a3841c98ff035dcf671060191538"
server_key_iv: "4876201f7e4da768248d1fa1cd8c9cda"
server_key_version: "encryptionkey@0"
I20260924 11:43:41.723063 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:41.735252 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:41.738672 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:41.738952 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:41.755541 13875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38935
I20260924 11:43:41.755585 13941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38935 every 8 connection(s)
I20260924 11:43:41.756759 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
I20260924 11:43:41.759968 13942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:41.765300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13875
I20260924 11:43:41.765484 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-1/wal/instance
I20260924 11:43:41.765992 388 external_mini_cluster.cc:1428] Setting key 54df89ae36b2d52977e54d3a4a333f12
I20260924 11:43:41.766177 13942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:43:41 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 11:43:41.787803 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.769115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.97.62:45393)
0924 11:43:41.769318 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:41.769326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:41.770239 (+ 913us) server_negotiation.cc:407] Connection header received
0924 11:43:41.772194 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:41.772202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:41.772305 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:41.772425 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:41.773916 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.774559 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.776837 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.777234 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.777336 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:41.781352 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:41.781361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:41.781369 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:41.781414 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:41.785097 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:41.785713 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:41.785719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:41.785722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:41.785882 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:41.786255 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:41.786260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:41.786263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:41.786628 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:41.786779 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:41.787354 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 11:43:41.787574 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:43:41.788610 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.769402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45393 (local address 127.0.0.1:49778)
0924 11:43:41.770063 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.770125 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:41.771928 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:41.772695 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:41.772702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:41.773140 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:41.773755 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:41.773781 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.774701 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.774709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:41.776705 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:41.776723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.777454 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.777463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:41.777773 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:41.778410 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:41.778440 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:41.781129 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:41.785220 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:41.785225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:41.785234 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:41.785589 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:41.785995 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:41.785998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:41.786000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:41.786146 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:41.786761 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:41.786766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:41.787019 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 11:43:41.787217 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 11:43:41.787558 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":293,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 11:43:41.790338 13942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:43:41 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:43:41.804102 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.791071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55920)
0924 11:43:41.791227 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.791245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:41.791360 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:41.792000 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:41.792004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:41.792465 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:41.792576 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:41.792592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.793863 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.793871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:41.794669 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:41.794678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.795388 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.795393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:41.795580 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:41.796112 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:41.796126 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:41.797926 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:41.801849 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:41.801852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:41.801855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:41.802200 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:41.802637 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:41.802640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:41.802642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:41.802781 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:41.803502 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:41.803505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:41.803602 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 11:43:41.803703 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:43:41.803863 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:43:41.804653 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.791167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.0.97.61:38935)
0924 11:43:41.791670 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:41.791675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:41.791696 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:43:41.791729 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:41.791733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:41.791868 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:41.792006 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:41.792728 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.793720 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.794787 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:41.795247 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:41.795374 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:41.798135 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:41.798149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:41.798165 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:41.798208 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:41.801729 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:41.802322 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:41.802332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:41.802338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:41.802471 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:41.802884 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:41.802892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:41.802898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:41.803366 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:41.803583 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:41.803880 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 11:43:41.804193 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260924 11:43:41.806819 13942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:43:41.808043 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.807498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51816)
0924 11:43:41.807689 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.807837 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:43:41.808359 13894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
I20260924 11:43:41.809753 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.809485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51822)
0924 11:43:41.809603 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.809672 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 11:43:41.809963 13942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 11:43:41.816995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46841
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45393,127.0.97.61:38935,127.0.97.60:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:43:41.859431 13942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 1
I20260924 11:43:41.860611 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.860002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51830)
0924 11:43:41.860255 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.860410 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:43:41.861738 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.861270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51840)
0924 11:43:41.861462 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.861572 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 11:43:41.862008 13942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
I20260924 11:43:41.930307 13942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 2
I20260924 11:43:41.931372 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.930895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51842)
0924 11:43:41.931101 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.931210 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:43:41.932415 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:41.931946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51844)
0924 11:43:41.932129 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:41.932255 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:43:41.932740 13942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
I20260924 11:43:42.008674 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } attempt: 5
I20260924 11:43:42.023936 13942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 3
I20260924 11:43:42.024996 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.024449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51860)
0924 11:43:42.024695 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.024810 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 11:43:42.025990 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.025591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51864)
0924 11:43:42.025805 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.025890 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260924 11:43:42.026268 13942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:43:42.033663 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.009598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.0.97.61:38935)
0924 11:43:42.009755 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.009759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.009827 (+ 68us) server_negotiation.cc:407] Connection header received
0924 11:43:42.009992 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.009996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.010063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.010141 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.011065 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.011673 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.013365 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.013783 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.013874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.028632 (+ 14758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.028636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.028639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.028668 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.031039 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.031797 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.031804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.031808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.031935 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.032354 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.032359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.032363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.032638 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.032793 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.033151 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.033380 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:43:42.033679 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.009321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55928)
0924 11:43:42.009720 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.009742 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:42.009880 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:42.010201 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:42.010206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:42.010709 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:42.010927 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.010939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.011821 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.011826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.013186 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.013205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.013927 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.013933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.014152 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.025873 (+ 11721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:42.025902 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:42.028454 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:42.031218 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.031223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.031227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.031689 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.032038 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.032045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.032050 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.032240 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.032759 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:42.032763 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:42.032920 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 11:43:42.033043 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 11:43:42.033335 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260924 11:43:42.034780 13868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:43:42.035689 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.035370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51870)
0924 11:43:42.035506 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.035575 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:43:42.036643 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.036343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51876)
0924 11:43:42.036470 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.036561 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:43:42.036856 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
I20260924 11:43:42.088129 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 1
I20260924 11:43:42.089212 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.088818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51888)
0924 11:43:42.088989 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.089069 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:43:42.090148 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.089855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51892)
0924 11:43:42.089999 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.090061 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:43:42.090327 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
W20260924 11:43:42.118096 13949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:42.118413 13949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:42.118561 13949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:42.125447 13949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:42.125584 13949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:42.125674 13949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:42.125790 13949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:42.125851 13949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:42.134971 13949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45393,127.0.97.61:38935,127.0.97.60:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:42.136034 13949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:42.137185 13949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:42.148243 13957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:42.148306 13955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:42.148351 13954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:42.148993 13949 server_base.cc:1034] running on GCE node
I20260924 11:43:42.149667 13949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:42.150766 13949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:42.151947 13949 hybrid_clock.cc:648] HybridClock initialized: now 1790250222151917 us; error 30 us; skew 500 ppm
I20260924 11:43:42.152323 13949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:42.155095 13949 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 11:43:42.156370 13949 webserver.cc:460] Webserver started at http://127.0.97.60:35907/ using document root <none> and password file <none>
I20260924 11:43:42.156904 13949 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:42.156988 13949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:42.157200 13949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:42.158546 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 2
I20260924 11:43:42.159612 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.159077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51902)
0924 11:43:42.159300 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.159394 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:43:42.160172 13949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data/instance:
uuid: "252186e9951e461d8bad3593b43f423c"
format_stamp: "Formatted at 2026-09-24 11:43:42 on dist-test-slave-qhrh"
server_key: "6d5d328a8575645e887ce4024f5f36ef"
server_key_iv: "c4f39c1693640108f9c82742237cacfd"
server_key_version: "encryptionkey@0"
I20260924 11:43:42.160662 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.160289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51918)
0924 11:43:42.160437 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.160511 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:43:42.160941 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
I20260924 11:43:42.161147 13949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal/instance:
uuid: "252186e9951e461d8bad3593b43f423c"
format_stamp: "Formatted at 2026-09-24 11:43:42 on dist-test-slave-qhrh"
server_key: "6d5d328a8575645e887ce4024f5f36ef"
server_key_iv: "c4f39c1693640108f9c82742237cacfd"
server_key_version: "encryptionkey@0"
I20260924 11:43:42.165720 13949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:43:42.168588 13964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:42.169471 13942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 4
I20260924 11:43:42.169732 13949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:43:42.169896 13949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal
uuid: "252186e9951e461d8bad3593b43f423c"
format_stamp: "Formatted at 2026-09-24 11:43:42 on dist-test-slave-qhrh"
server_key: "6d5d328a8575645e887ce4024f5f36ef"
server_key_iv: "c4f39c1693640108f9c82742237cacfd"
server_key_version: "encryptionkey@0"
I20260924 11:43:42.170095 13949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:42.170475 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.169978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51932)
0924 11:43:42.170205 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.170306 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:43:42.171397 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.171101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51942)
0924 11:43:42.171232 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.171307 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 11:43:42.171669 13942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46841: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46841: connect: Connection refused (error 111)
I20260924 11:43:42.185520 13949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:42.188781 13949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:42.188966 13949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:42.205410 13949 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46841
I20260924 11:43:42.205480 14015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46841 every 8 connection(s)
I20260924 11:43:42.206492 13949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
I20260924 11:43:42.209604 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:42.214865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 13949
I20260924 11:43:42.215061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250027667721-388-0/minicluster-data/master-2/wal/instance
I20260924 11:43:42.215580 388 external_mini_cluster.cc:1428] Setting key 477718a0af5f4e74a256ce2865751cc5
I20260924 11:43:42.215720 14016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:43:42.234994 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.218300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.0.97.62:45393)
0924 11:43:42.218523 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.218531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.219021 (+ 490us) server_negotiation.cc:407] Connection header received
0924 11:43:42.220135 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.220159 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.220236 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.220336 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.221965 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.222908 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.224770 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.225151 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.225242 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.228839 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.228847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.228854 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.228896 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.231884 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.232671 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.232676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.232680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.232819 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.233209 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.233218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.233222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.233576 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.233692 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.234528 (+ 836us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.234667 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:43:42.235690 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.218276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45393 (local address 127.0.0.1:49792)
0924 11:43:42.218894 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.218930 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:42.219909 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:42.220399 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:42.220406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:42.221198 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:42.221797 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.221826 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.223057 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.223065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.224643 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.224657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.225381 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.225394 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.225560 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.226188 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:42.226215 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:42.228651 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:42.232022 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.232030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.232045 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.232559 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.232929 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.232932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.232934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.233093 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.233706 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:42.233713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:42.234086 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 11:43:42.234349 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 11:43:42.234777 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":309,"threads_started":1}
I20260924 11:43:42.236338 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.217254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.0.97.60:46841)
0924 11:43:42.217944 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.217957 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.217995 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:43:42.219898 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.219916 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.220122 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.220363 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.222271 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.223617 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.225381 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.226000 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.226117 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.230247 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.230256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.230268 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.230303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.233447 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.234202 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.234208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.234210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.234354 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.234781 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.234787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.234791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.235172 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.235441 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.235746 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.235949 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":309,"threads_started":1}
I20260924 11:43:42.237582 14016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:42.249707 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.238208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38935 (local address 127.0.0.1:55930)
0924 11:43:42.238340 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.238355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:42.238470 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:42.238982 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:42.238985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:42.239266 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:42.239378 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.239385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.240264 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.240270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.241003 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.241013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.241818 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.241824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.242017 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.242491 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:42.242510 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:42.244682 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:42.247614 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.247619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.247622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.247926 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.248322 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.248326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.248329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.248446 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.248894 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:42.248898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:42.249042 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 11:43:42.249162 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 11:43:42.249394 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:43:42.249755 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.238186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.0.97.61:38935)
0924 11:43:42.238324 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.238329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.238427 (+ 98us) server_negotiation.cc:407] Connection header received
0924 11:43:42.238604 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.238608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.238700 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.238793 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.239519 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.240109 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.241144 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.241673 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.241791 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.244844 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.244848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.244850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.244877 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.247480 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.248030 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.248045 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.248052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.248218 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.248541 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.248548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.248552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.248786 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.248934 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.249266 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.249445 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 11:43:42.250682 14016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:43:42.251416 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.238841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.97.62:45393)
0924 11:43:42.239029 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.239033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.239059 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:43:42.239098 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.239101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.239139 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.239243 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.240329 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.241042 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.242496 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.242851 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.242935 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.246684 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.246688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.246691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.246717 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.248943 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.249531 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.249537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.249539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.249645 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.250051 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.250055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.250058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.250298 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.250471 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.250961 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.251141 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:43:42.263746 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.251372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51960)
0924 11:43:42.251488 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.251504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:42.251587 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:42.251910 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:42.251914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:42.252330 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:42.252434 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.252441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.253635 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.253638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.254488 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.254501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.255127 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.255133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.255294 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.255878 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:42.255893 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:42.258133 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:42.261368 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.261375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.261380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.261840 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.262214 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.262221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.262226 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.262343 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.262824 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:42.262829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:42.262953 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 11:43:42.263097 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:43:42.263363 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 11:43:42.263804 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.251558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.0.97.60:46841)
0924 11:43:42.251689 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.251694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.251708 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:43:42.251745 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.251752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.251811 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.251896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.252559 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.253500 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.254626 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.255000 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.255088 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.258303 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.258309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.258313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.258351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.261217 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.262003 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.262006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.262008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.262112 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.262451 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.262455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.262459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.262706 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.262843 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.263177 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.263487 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":35}
I20260924 11:43:42.266968 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.252756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.97.61:38935)
0924 11:43:42.252890 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.252893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.252933 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:43:42.253086 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.253088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.253128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.253191 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.254456 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.255256 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.256512 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.257149 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.257237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.262007 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.262012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.262014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.262050 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.264717 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.265272 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.265279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.265285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.265421 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.265785 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.265791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.265794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.266099 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.266359 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.266491 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.266674 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:43:42.267204 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 3
I20260924 11:43:42.269243 14016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: Bootstrap starting.
I20260924 11:43:42.271390 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:43:42.273212 14016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: Neither blocks nor log segments found. Creating new log.
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:43:42.274601 14016 log.cc:824] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:42.278828 14016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: No bootstrap required, opened a new log
I20260924 11:43:42.281873 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.267683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51970)
0924 11:43:42.267904 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.267920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:42.268046 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:42.268356 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:42.268362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:42.268835 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:42.269016 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.269027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.270263 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.270267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.271216 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.271227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.271986 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.271991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.272187 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.272791 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:42.272807 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:42.274959 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:42.279096 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.279100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.279103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.279506 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.279942 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.279947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.279951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.280114 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.280861 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:42.280866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:42.281053 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 11:43:42.281205 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 11:43:42.281504 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 11:43:42.282230 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.267734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.0.97.60:46841)
0924 11:43:42.267891 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.267896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.267990 (+ 94us) server_negotiation.cc:407] Connection header received
0924 11:43:42.268181 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.268188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.268251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.268331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.269170 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.270099 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.271359 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.271861 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.272020 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.275123 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.275128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.275132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.275169 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.278955 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.279647 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.279652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.279656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.279813 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.280245 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.280251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.280257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.280708 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.280925 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.281379 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.281697 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 11:43:42.288668 13868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: Bootstrap starting.
I20260924 11:43:42.291002 14016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.291424 14016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:42.291559 14016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252186e9951e461d8bad3593b43f423c, State: Initialized, Role: FOLLOWER
I20260924 11:43:42.292317 14016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.293780 13868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:42.295285 13868 log.cc:824] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:42.299289 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.299662 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.301465 14016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:42.311983 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.280502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.0.97.62:45393)
0924 11:43:42.280650 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.280655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.290808 (+ 10153us) server_negotiation.cc:407] Connection header received
0924 11:43:42.291241 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.291245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.291302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.291361 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.296395 (+ 5034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.297625 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.300715 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.301589 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.301702 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.305681 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.305689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.305696 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.305731 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.309014 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.309914 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.309920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.309923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.310055 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.310410 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.310415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.310418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.310722 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.310952 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.311453 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.311662 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:43:42.321457 13868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: No bootstrap required, opened a new log
I20260924 11:43:42.335389 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.283717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.0.97.61:38935)
0924 11:43:42.283888 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.283896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.303127 (+ 19231us) server_negotiation.cc:407] Connection header received
0924 11:43:42.303365 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.303372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.303456 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.303527 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.305570 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.306634 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.323680 (+ 17046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.324402 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.324515 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.328098 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.328104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.328108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.328160 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.331950 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.333110 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.333115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.333119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.333258 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.333693 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.333698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.333701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.334051 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.334282 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.334808 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.335021 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:43:42.336331 13868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.336683 13868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:42.336786 13868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0694a96692f64996881b89e1892da9f3, State: Initialized, Role: FOLLOWER
I20260924 11:43:42.337432 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.292476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.97.60:46841)
0924 11:43:42.292637 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.292643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.312780 (+ 20137us) server_negotiation.cc:407] Connection header received
0924 11:43:42.312997 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.313002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.313075 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.313144 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.315036 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.316174 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.317943 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.318562 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.318644 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.330551 (+ 11907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.330559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.330566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.330602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.333528 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.334366 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.334374 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.334379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.334553 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.335088 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.335095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.335100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.335529 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.335814 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.336736 (+ 922us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.336984 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:43:42.337432 13868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.338343 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.338677 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.339504 13868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:43:42.349164 14047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:43:42.349305 14047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:43:42.346920 13949 master_runner.cc:186] Error getting master registration for 127.0.97.62:45393: OK, instance_id { permanent_uuid: "0694a96692f64996881b89e1892da9f3" instance_seqno: 1790250221303281 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:43:42.356590 13949 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 11:43:42.364075 14054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:43:42.364276 14054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:43:42.356643 13801 master_runner.cc:186] Error getting master registration for 127.0.97.61:38935: OK, instance_id { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" instance_seqno: 1790250221750791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:43:42.367952 13801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 11:43:42.440711 13970 tablet.cc:2369] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:42.470896 13942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } attempt: 5
Sep 24 11:43:42 dist-test-slave-qhrh krb5kdc[13788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:43:42.487771 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.471443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46841 (local address 127.0.0.1:51996)
0924 11:43:42.471844 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:42.471864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:42.471975 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:42.472697 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:42.472702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:42.473209 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:42.473473 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.473497 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.474950 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.474955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.476319 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:42.476333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.477311 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.477317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:42.477502 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.478165 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:42.478188 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:42.480869 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:42.484954 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.484960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.484964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.485474 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.485876 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:42.485883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:42.485887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:42.486061 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:42.486778 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:42.486783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:42.486981 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 11:43:42.487143 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 11:43:42.487392 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260924 11:43:42.487911 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:42.471643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.97.60:46841)
0924 11:43:42.472221 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:42.472227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:42.472249 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:43:42.472310 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:42.472315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:42.472384 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:42.472475 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:42.473670 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.474801 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.476464 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:42.477160 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:42.477279 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:42.481081 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:42.481088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:42.481092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:42.481130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:42.484827 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.485612 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.485617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.485621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.485751 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:42.486212 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:42.486218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:42.486222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:42.486632 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:42.486805 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:42.487256 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 11:43:42.487508 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260924 11:43:42.494535 13942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b: Bootstrap starting.
I20260924 11:43:42.498396 13942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:42.499542 13942 log.cc:824] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:42.502730 13942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b: No bootstrap required, opened a new log
I20260924 11:43:42.513859 13942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.514232 13942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:42.514348 13942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f183e831dff48c39464cb48e953d40b, State: Initialized, Role: FOLLOWER
I20260924 11:43:42.514937 13942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.515817 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.516276 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.516871 13942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:42.532832 13875 master_runner.cc:263] No leader master found from master 0f183e831dff48c39464cb48e953d40b
I20260924 11:43:42.532965 13875 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 11:43:42.538498 14068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:43:42.538621 14068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 11:43:42.542878 13822 tablet.cc:2369] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:42.626919 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:43:42.627297 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.629016 14057 leader_election.cc:288] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0694a96692f64996881b89e1892da9f3 (127.0.97.62:45393), 252186e9951e461d8bad3593b43f423c (127.0.97.60:46841)
I20260924 11:43:42.629982 13843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f183e831dff48c39464cb48e953d40b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0694a96692f64996881b89e1892da9f3" is_pre_election: true
I20260924 11:43:42.630595 13843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f183e831dff48c39464cb48e953d40b in term 0.
I20260924 11:43:42.631179 13891 leader_election.cc:302] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0694a96692f64996881b89e1892da9f3, 0f183e831dff48c39464cb48e953d40b; no voters:
I20260924 11:43:42.632133 13991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f183e831dff48c39464cb48e953d40b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252186e9951e461d8bad3593b43f423c" is_pre_election: true
I20260924 11:43:42.632741 13991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f183e831dff48c39464cb48e953d40b in term 0.
I20260924 11:43:42.633582 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:43:42.633742 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:43:42.633852 14057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:42.637485 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.638453 13843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f183e831dff48c39464cb48e953d40b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0694a96692f64996881b89e1892da9f3"
I20260924 11:43:42.638671 13843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:42.642174 13843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f183e831dff48c39464cb48e953d40b in term 1.
I20260924 11:43:42.646723 13891 leader_election.cc:302] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0694a96692f64996881b89e1892da9f3, 0f183e831dff48c39464cb48e953d40b; no voters:
I20260924 11:43:42.650328 13991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f183e831dff48c39464cb48e953d40b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252186e9951e461d8bad3593b43f423c"
I20260924 11:43:42.650550 13991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:42.653921 13991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f183e831dff48c39464cb48e953d40b in term 1.
I20260924 11:43:42.654590 14057 leader_election.cc:288] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [CANDIDATE]: Term 1 election: Requested vote from peers 0694a96692f64996881b89e1892da9f3 (127.0.97.62:45393), 252186e9951e461d8bad3593b43f423c (127.0.97.60:46841)
I20260924 11:43:42.654793 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:42.654927 14057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [term 1 LEADER]: Becoming Leader. State: Replica: 0f183e831dff48c39464cb48e953d40b, State: Running, Role: LEADER
I20260924 11:43:42.655422 14057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } }
I20260924 11:43:42.664551 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f183e831dff48c39464cb48e953d40b. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.664867 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:42.666905 14072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:42.669190 14072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:42.687827 13991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [term 1 FOLLOWER]: Refusing update from remote peer 0f183e831dff48c39464cb48e953d40b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:42.688761 14057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [LEADER]: Connected to new peer: Peer: permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:42.695693 13843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [term 1 FOLLOWER]: Refusing update from remote peer 0f183e831dff48c39464cb48e953d40b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:42.696518 14057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:42.732810 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f183e831dff48c39464cb48e953d40b. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.733106 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.737322 14073 mvcc.cc:205] Tried to move back new op lower bound from 7332864912118910976 to 7332864912008065024. Current Snapshot: MvccSnapshot[applied={T|T < 7332864912118910976}]
I20260924 11:43:42.740566 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.740971 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:42.741394 14075 mvcc.cc:205] Tried to move back new op lower bound from 7332864912118910976 to 7332864912008065024. Current Snapshot: MvccSnapshot[applied={T|T < 7332864912118910976}]
I20260924 11:43:42.742435 14072 catalog_manager.cc:1347] Generated new cluster ID: 7ed9522fb76b4f84a4b6805b690eb847
I20260924 11:43:42.742568 14072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:43:42.748427 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.748642 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.758097 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f183e831dff48c39464cb48e953d40b. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.758397 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.760973 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.761245 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:42.764006 14074 mvcc.cc:205] Tried to move back new op lower bound from 7332864912118910976 to 7332864912008065024. Current Snapshot: MvccSnapshot[applied={T|T < 7332864912118910976}]
I20260924 11:43:42.775385 14072 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:43:42.777017 14072 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:43:42.777113 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f183e831dff48c39464cb48e953d40b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0694a96692f64996881b89e1892da9f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } peers { permanent_uuid: "0f183e831dff48c39464cb48e953d40b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38935 } } peers { permanent_uuid: "252186e9951e461d8bad3593b43f423c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46841 } } }
I20260924 11:43:42.777330 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:43:42.810571 14072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f183e831dff48c39464cb48e953d40b: Generated new TSK 0
I20260924 11:43:42.811542 14072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:43:43.351230 14047 catalog_manager.cc:1259] Loaded cluster ID: 7ed9522fb76b4f84a4b6805b690eb847
I20260924 11:43:43.351322 14047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: loading cluster ID for follower catalog manager: success
I20260924 11:43:43.353581 14047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: acquiring CA information for follower catalog manager: success
I20260924 11:43:43.355201 14047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 252186e9951e461d8bad3593b43f423c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:43:43.365789 14054 catalog_manager.cc:1259] Loaded cluster ID: 7ed9522fb76b4f84a4b6805b690eb847
I20260924 11:43:43.365862 14054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: loading cluster ID for follower catalog manager: success
I20260924 11:43:43.367820 14054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: acquiring CA information for follower catalog manager: success
I20260924 11:43:43.369278 14054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0694a96692f64996881b89e1892da9f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:43:43.376098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13801
I20260924 11:43:43.377167 13863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:43.480443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13801
I20260924 11:43:43.500336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13875
I20260924 11:43:43.501430 13937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:43.601836 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13875
I20260924 11:43:43.621703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 13949
I20260924 11:43:43.622697 14011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:43.722009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 13949
2026-09-24T11:43:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4028 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:43:43 dist-test-slave-qhrh krb5kdc[14090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:43:43 dist-test-slave-qhrh krb5kdc[14090](info): set up 2 sockets
Sep 24 11:43:43 dist-test-slave-qhrh krb5kdc[14090](info): commencing operation
krb5kdc: starting...
W20260924 11:43:44.744441 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:43:44 dist-test-slave-qhrh krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:43:44.935192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40579
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:45.238683 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:45.238946 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:45.239022 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:45.245730 14103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:45.245810 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:45.245850 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:45.245896 14103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:45.245973 14103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:45.254534 14103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:45.255609 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:45.256819 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:45.267542 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:45.267488 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:45.267577 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:45.268198 14103 server_base.cc:1034] running on GCE node
I20260924 11:43:45.268894 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:45.270105 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:45.271294 14103 hybrid_clock.cc:648] HybridClock initialized: now 1790250225271256 us; error 36 us; skew 500 ppm
I20260924 11:43:45.271601 14103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:45 dist-test-slave-qhrh krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:45.274423 14103 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:45.275765 14103 webserver.cc:460] Webserver started at http://127.0.97.62:44147/ using document root <none> and password file <none>
I20260924 11:43:45.276432 14103 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:45.276564 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:45.276825 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:45.279263 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "fdd50d3186524c2ca20b71da6f72f9c4"
format_stamp: "Formatted at 2026-09-24 11:43:45 on dist-test-slave-qhrh"
server_key: "6d9515c1e85121ee62ed44441aaa91ab"
server_key_iv: "5194f0cb2d3b27fc52fbd8943141d292"
server_key_version: "encryptionkey@0"
I20260924 11:43:45.279932 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "fdd50d3186524c2ca20b71da6f72f9c4"
format_stamp: "Formatted at 2026-09-24 11:43:45 on dist-test-slave-qhrh"
server_key: "6d9515c1e85121ee62ed44441aaa91ab"
server_key_iv: "5194f0cb2d3b27fc52fbd8943141d292"
server_key_version: "encryptionkey@0"
I20260924 11:43:45.284595 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:43:45.287799 14118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:45.289023 14103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:43:45.289183 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "fdd50d3186524c2ca20b71da6f72f9c4"
format_stamp: "Formatted at 2026-09-24 11:43:45 on dist-test-slave-qhrh"
server_key: "6d9515c1e85121ee62ed44441aaa91ab"
server_key_iv: "5194f0cb2d3b27fc52fbd8943141d292"
server_key_version: "encryptionkey@0"
I20260924 11:43:45.289387 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:45.300096 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:45.303725 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:45.303994 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:45.321108 14103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40579
I20260924 11:43:45.321146 14169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40579 every 8 connection(s)
I20260924 11:43:45.322465 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:45.325841 14170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:45.332443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14103
I20260924 11:43:45.332667 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:43:45.333425 388 external_mini_cluster.cc:1428] Setting key 47bf3febc27b0bc448c76e6e3080bb81
I20260924 11:43:45.336226 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4: Bootstrap starting.
I20260924 11:43:45.339267 14170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:45.340368 14170 log.cc:824] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:45.343230 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4: No bootstrap required, opened a new log
Sep 24 11:43:45 dist-test-slave-qhrh krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:45.352844 14170 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40579 } }
I20260924 11:43:45.353204 14170 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:45.353317 14170 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd50d3186524c2ca20b71da6f72f9c4, State: Initialized, Role: FOLLOWER
I20260924 11:43:45.353927 14170 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40579 } }
I20260924 11:43:45.354231 14170 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:43:45.354379 14170 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:43:45.354555 14170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:45.355994 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:45.335146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.0.97.62:40579)
0924 11:43:45.335814 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:45.335824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:45.335864 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:43:45.337528 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:45.337547 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:45.337871 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:45.338156 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:45.339761 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:45.341309 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:45.343095 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:45.343781 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:45.343897 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:45.348130 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:45.348162 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:45.348177 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:45.348213 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:45.351899 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:45.352514 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:45.352521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:45.352528 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:45.352707 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:45.353025 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:45.353030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:45.353032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:45.353691 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:45.353985 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:45.354288 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:43:45.354713 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
I20260924 11:43:45.357636 14170 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40579 } }
I20260924 11:43:45.358137 14170 leader_election.cc:302] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd50d3186524c2ca20b71da6f72f9c4; no voters:
I20260924 11:43:45.358848 14170 leader_election.cc:288] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:43:45.358933 14175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:45.359934 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:43:45.360420 14175 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [term 1 LEADER]: Becoming Leader. State: Replica: fdd50d3186524c2ca20b71da6f72f9c4, State: Running, Role: LEADER
I20260924 11:43:45.361063 14170 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:45.361068 14175 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40579 } }
I20260924 11:43:45.364995 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd50d3186524c2ca20b71da6f72f9c4. Latest consensus state: current_term: 1 leader_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40579 } } }
I20260924 11:43:45.365343 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:45.365808 14176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd50d3186524c2ca20b71da6f72f9c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40579 } } }
I20260924 11:43:45.366118 14176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:45.367005 14184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:45.371604 14184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:45.379616 14184 catalog_manager.cc:1347] Generated new cluster ID: 467d96935794431bbd667a1fad32d6a3
I20260924 11:43:45.379724 14184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:43:45.389730 14184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:43:45.391319 14184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:43:45.405479 14184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd50d3186524c2ca20b71da6f72f9c4: Generated new TSK 0
I20260924 11:43:45.406260 14184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:43:45.421452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14103
I20260924 11:43:45.422528 14165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:45.507085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14103
2026-09-24T11:43:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1780 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:43:45 dist-test-slave-qhrh krb5kdc[14198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:43:45 dist-test-slave-qhrh krb5kdc[14198](info): set up 2 sockets
Sep 24 11:43:45 dist-test-slave-qhrh krb5kdc[14198](info): commencing operation
krb5kdc: starting...
W20260924 11:43:46.528584 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:43:46 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:43:46.719733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35925
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:47.017318 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:47.017597 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:47.017755 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:47.024430 14211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:47.024554 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:47.024649 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:47.024748 14211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:47.024840 14211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:47.033907 14211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:47.035053 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:47.036228 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:47.046850 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:47.046914 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:47.046911 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:47.047420 14211 server_base.cc:1034] running on GCE node
I20260924 11:43:47.048211 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:47.049417 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:47.050619 14211 hybrid_clock.cc:648] HybridClock initialized: now 1790250227050583 us; error 32 us; skew 500 ppm
I20260924 11:43:47.050889 14211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:47 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:47.072945 14211 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:47.074410 14211 webserver.cc:460] Webserver started at http://127.0.97.62:44993/ using document root <none> and password file <none>
I20260924 11:43:47.075029 14211 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:47.075130 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:47.075361 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:47.078147 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "2f30d11e5e45413db6dda133ac2004b8"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "957e4840b73ff46f3b9f260d90265cf5"
server_key_iv: "390b237bac74d28dc8f0fa762eb18d65"
server_key_version: "encryptionkey@0"
I20260924 11:43:47.078858 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f30d11e5e45413db6dda133ac2004b8"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "957e4840b73ff46f3b9f260d90265cf5"
server_key_iv: "390b237bac74d28dc8f0fa762eb18d65"
server_key_version: "encryptionkey@0"
I20260924 11:43:47.083427 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 11:43:47.086494 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:47.087639 14211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:43:47.087802 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "2f30d11e5e45413db6dda133ac2004b8"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "957e4840b73ff46f3b9f260d90265cf5"
server_key_iv: "390b237bac74d28dc8f0fa762eb18d65"
server_key_version: "encryptionkey@0"
I20260924 11:43:47.087999 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:47.109577 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:47.113034 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:47.113296 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:47.130878 14211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35925
I20260924 11:43:47.130952 14277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35925 every 8 connection(s)
I20260924 11:43:47.132072 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:47.135390 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:47.137257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14211
I20260924 11:43:47.137569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:43:47.138183 388 external_mini_cluster.cc:1428] Setting key bf54626a9d15de4511b50c27ba0c76df
I20260924 11:43:47.147454 14278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap starting.
I20260924 11:43:47.150616 14278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:47.151463 14278 log.cc:824] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:43:47 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:47.154353 14278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: No bootstrap required, opened a new log
I20260924 11:43:47.161016 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:47.140099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.0.97.62:35925)
0924 11:43:47.140875 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:47.140888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:47.140935 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:43:47.143216 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:47.143239 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:47.143713 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:47.144123 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:47.145406 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:47.147197 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:47.148515 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:47.149541 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:47.149718 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:47.153332 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:47.153341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:47.153357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:47.153396 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:47.157063 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:47.157644 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:47.157651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:47.157658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:47.157843 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:47.158163 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:47.158167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:47.158170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:47.158729 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:47.158994 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:47.159286 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 11:43:47.159684 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":288,"threads_started":1}
I20260924 11:43:47.163661 14278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:47.163944 14278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:47.164012 14278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f30d11e5e45413db6dda133ac2004b8, State: Initialized, Role: FOLLOWER
I20260924 11:43:47.164505 14278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:47.164686 14278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:43:47.164784 14278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:43:47.164896 14278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:47.166709 14278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:47.167163 14278 leader_election.cc:302] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f30d11e5e45413db6dda133ac2004b8; no voters:
I20260924 11:43:47.167821 14278 leader_election.cc:288] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:43:47.167945 14283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:47.168686 14283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2f30d11e5e45413db6dda133ac2004b8, State: Running, Role: LEADER
I20260924 11:43:47.169195 14283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:47.169711 14278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:47.172631 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f30d11e5e45413db6dda133ac2004b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } } }
I20260924 11:43:47.173020 14285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f30d11e5e45413db6dda133ac2004b8. Latest consensus state: current_term: 1 leader_uuid: "2f30d11e5e45413db6dda133ac2004b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } } }
I20260924 11:43:47.173779 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:47.173796 14285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:47.174893 14292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:47.179229 14292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:47.187805 14292 catalog_manager.cc:1347] Generated new cluster ID: 40f269c04f714abd949f3dac8c9b399f
I20260924 11:43:47.187906 14292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:43:47.210515 14292 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:43:47.211565 14292 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:43:47.217412 14292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Generated new TSK 0
I20260924 11:43:47.218014 14292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:43:47.275164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35925
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:47.579888 14304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:47.580310 14304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:47.580499 14304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:47.588307 14304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:47.588439 14304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:47.588624 14304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:43:47.597918 14304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:47.598984 14304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:47.600253 14304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:47.612624 14310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:47.612648 14309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:47.612586 14312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:47.613348 14304 server_base.cc:1034] running on GCE node
I20260924 11:43:47.613929 14304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:47.614737 14304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:47.615921 14304 hybrid_clock.cc:648] HybridClock initialized: now 1790250227615889 us; error 37 us; skew 500 ppm
I20260924 11:43:47.616256 14304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:47 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:47.619201 14304 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:43:47.620476 14304 webserver.cc:460] Webserver started at http://127.0.97.1:44253/ using document root <none> and password file <none>
I20260924 11:43:47.621088 14304 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:47.621220 14304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:47.621490 14304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:47.624045 14304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "62055ee072fa40b5ba835375ee3233e2"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "b20f0016f3baa7842166b78d99660885"
server_key_iv: "936cac53dff84e69af2a556fa4060e66"
server_key_version: "encryptionkey@0"
I20260924 11:43:47.624898 14304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "62055ee072fa40b5ba835375ee3233e2"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "b20f0016f3baa7842166b78d99660885"
server_key_iv: "936cac53dff84e69af2a556fa4060e66"
server_key_version: "encryptionkey@0"
I20260924 11:43:47.629455 14304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:43:47.632627 14319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:47.633760 14304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:43:47.633921 14304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "62055ee072fa40b5ba835375ee3233e2"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "b20f0016f3baa7842166b78d99660885"
server_key_iv: "936cac53dff84e69af2a556fa4060e66"
server_key_version: "encryptionkey@0"
I20260924 11:43:47.634109 14304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:47.654304 14304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:47.657588 14304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:47.657842 14304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:47.658648 14304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:47.659828 14304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:47.659915 14304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:47.660060 14304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:47.660122 14304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:47.690125 14304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33745
I20260924 11:43:47.690187 14431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33745 every 8 connection(s)
I20260924 11:43:47.691212 14304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:43:47.693446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14304
I20260924 11:43:47.693639 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:43:47.694504 388 external_mini_cluster.cc:1428] Setting key 98252a3cd9908dae0b4c9da7b34c22af
Sep 24 11:43:47 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:47.711604 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:47.693548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57043 (local address 127.0.97.62:35925)
0924 11:43:47.694013 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:47.694018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:47.694983 (+ 965us) server_negotiation.cc:407] Connection header received
0924 11:43:47.697062 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:47.697068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:47.697137 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:47.697195 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:47.698662 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:47.699425 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:47.701002 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:47.701408 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:47.701539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:47.705197 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:47.705205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:47.705212 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:47.705269 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:47.708434 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:47.709112 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:47.709120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:47.709126 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:47.709281 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:47.709691 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:47.709698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:47.709704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:47.710079 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:47.710221 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:47.710818 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 11:43:47.711131 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":261,"threads_started":1}
I20260924 11:43:47.711975 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:47.694101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:57043)
0924 11:43:47.694822 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:47.694872 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:47.696677 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:47.697547 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:47.697554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:47.698007 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:47.698494 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:47.698509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:47.699578 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:47.699583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:47.700866 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:47.700877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:47.701630 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:47.701640 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:47.701852 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:47.702487 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:47.702515 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:47.705015 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:47.708553 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:47.708559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:47.708568 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:47.708961 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:47.709396 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:47.709399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:47.709401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:47.709564 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:47.710190 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:47.710195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:47.710443 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 11:43:47.710670 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 11:43:47.711031 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":304,"threads_started":1}
I20260924 11:43:47.713394 14432 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35925
I20260924 11:43:47.713682 14432 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:47.714315 14432 heartbeater.cc:511] Master 127.0.97.62:35925 requested a full tablet report, sending...
I20260924 11:43:47.716205 14243 ts_manager.cc:194] Registered new tserver with Master: 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 11:43:47.717679 14243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:43:47.742568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35925
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:48.033901 14438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:48.034132 14438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:48.034274 14438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:48.040840 14438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:48.040938 14438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:48.041085 14438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:43:48.049674 14438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:48.050683 14438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:48.051842 14438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:48.063917 14444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:48.063971 14443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:48.064021 14446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:48.064770 14438 server_base.cc:1034] running on GCE node
I20260924 11:43:48.065299 14438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:48.066097 14438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:48.067323 14438 hybrid_clock.cc:648] HybridClock initialized: now 1790250228067290 us; error 52 us; skew 500 ppm
I20260924 11:43:48.067606 14438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:48 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:48.070564 14438 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:43:48.071767 14438 webserver.cc:460] Webserver started at http://127.0.97.2:45043/ using document root <none> and password file <none>
I20260924 11:43:48.072364 14438 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:48.072453 14438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:48.072682 14438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:48.075225 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "c698cf01febc437bba468e3f996726f1"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "fd52279d8d26d22b9c66e4a3d8d3aebf"
server_key_iv: "1ecbac803d7d99a24924ef4cafc952c6"
server_key_version: "encryptionkey@0"
I20260924 11:43:48.075997 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c698cf01febc437bba468e3f996726f1"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "fd52279d8d26d22b9c66e4a3d8d3aebf"
server_key_iv: "1ecbac803d7d99a24924ef4cafc952c6"
server_key_version: "encryptionkey@0"
I20260924 11:43:48.080634 14438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:43:48.083523 14453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:48.084735 14438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:43:48.084905 14438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "c698cf01febc437bba468e3f996726f1"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "fd52279d8d26d22b9c66e4a3d8d3aebf"
server_key_iv: "1ecbac803d7d99a24924ef4cafc952c6"
server_key_version: "encryptionkey@0"
I20260924 11:43:48.085081 14438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:48.101975 14438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:48.105842 14438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:48.106093 14438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:48.106861 14438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:48.107977 14438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:48.108042 14438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:48.108210 14438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:48.108280 14438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:48.138494 14438 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45799
I20260924 11:43:48.138566 14565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45799 every 8 connection(s)
I20260924 11:43:48.139523 14438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:43:48.149816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14438
I20260924 11:43:48.149974 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:43:48.150409 388 external_mini_cluster.cc:1428] Setting key d7780db7a70cf801b64cce89f2f98495
Sep 24 11:43:48 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:48.157310 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.142058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38135 (local address 127.0.97.62:35925)
0924 11:43:48.142221 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.142225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.143325 (+ 1100us) server_negotiation.cc:407] Connection header received
0924 11:43:48.145104 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.145109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.145203 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.145303 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:48.146522 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.147158 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.148529 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.148894 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.149021 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.152121 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:48.152129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:48.152179 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:48.152222 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:48.154664 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:48.155313 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:48.155320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:48.155325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:48.155455 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:48.155828 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:48.155833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:48.155836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:48.156074 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:48.156235 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.156790 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.157009 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:43:48.158066 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.142570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:38135)
0924 11:43:48.143193 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.143230 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.144825 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.145518 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.145524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.145937 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:48.146373 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.146387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.147280 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.147285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.148395 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.148405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.149108 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.149115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.149301 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.149858 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:48.149912 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:48.151909 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:48.154811 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:48.154830 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:48.154843 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:48.155204 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:48.155556 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:48.155561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:48.155564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:48.155727 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:48.156214 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:48.156220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:48.156499 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.156722 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.157139 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":269,"threads_started":1}
I20260924 11:43:48.159271 14566 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35925
I20260924 11:43:48.159566 14566 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:48.160214 14566 heartbeater.cc:511] Master 127.0.97.62:35925 requested a full tablet report, sending...
I20260924 11:43:48.161614 14243 ts_manager.cc:194] Registered new tserver with Master: c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799)
I20260924 11:43:48.162370 14243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38135
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:43:48.195287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35925
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:48.483304 14571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:48.483572 14571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:48.483749 14571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:48.490509 14571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:48.490638 14571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:48.490854 14571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:43:48.499614 14571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:48.500671 14571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:48.501920 14571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:48.513579 14576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:48.513618 14579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:48.513550 14577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:48.514346 14571 server_base.cc:1034] running on GCE node
I20260924 11:43:48.514904 14571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:48.515751 14571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:48.516955 14571 hybrid_clock.cc:648] HybridClock initialized: now 1790250228516905 us; error 43 us; skew 500 ppm
I20260924 11:43:48.517222 14571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:48 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:48.520020 14571 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:43:48.521370 14571 webserver.cc:460] Webserver started at http://127.0.97.3:38175/ using document root <none> and password file <none>
I20260924 11:43:48.521921 14571 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:48.522013 14571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:48.522238 14571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:43:48.524986 14571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "a9aa9bc6555a461cdcaa2d623139758a"
server_key_iv: "6f0253b4598f7e2e7c40f84c30da3140"
server_key_version: "encryptionkey@0"
I20260924 11:43:48.525652 14571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "a9aa9bc6555a461cdcaa2d623139758a"
server_key_iv: "6f0253b4598f7e2e7c40f84c30da3140"
server_key_version: "encryptionkey@0"
I20260924 11:43:48.529965 14571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:43:48.533002 14586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:48.534152 14571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:43:48.534271 14571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "a9aa9bc6555a461cdcaa2d623139758a"
server_key_iv: "6f0253b4598f7e2e7c40f84c30da3140"
server_key_version: "encryptionkey@0"
I20260924 11:43:48.534415 14571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:48.549548 14571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:48.552924 14571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:48.553114 14571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:48.553930 14571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:48.555075 14571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:48.555145 14571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:48.555243 14571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:48.555285 14571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:48.586796 14571 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33125
I20260924 11:43:48.586833 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33125 every 8 connection(s)
I20260924 11:43:48.587966 14571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:43:48.592519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14571
I20260924 11:43:48.592684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:43:48.593101 388 external_mini_cluster.cc:1428] Setting key 8380b1ec7f706c36f68007481b135fa0
Sep 24 11:43:48 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:48.606377 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.590356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41039 (local address 127.0.97.62:35925)
0924 11:43:48.590551 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.590557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.591835 (+ 1278us) server_negotiation.cc:407] Connection header received
0924 11:43:48.593898 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.593904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.593996 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.594086 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:48.595929 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.596719 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.598050 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.598391 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.598471 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.601603 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:48.601608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:48.601611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:48.601640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:48.603976 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:48.604614 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:48.604618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:48.604620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:48.604731 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:48.605085 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:48.605088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:48.605090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:48.605329 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:48.605435 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.605967 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.606108 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:43:48.607308 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.590892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:41039)
0924 11:43:48.591651 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.591720 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.593583 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.594362 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.594372 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.595010 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:48.595763 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.595791 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.596844 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.596848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.597944 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.597954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.598546 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.598583 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.598816 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.599429 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:48.599450 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:48.601424 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:48.604088 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:48.604094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:48.604103 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:48.604520 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:48.604821 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:48.604824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:48.604826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:48.604991 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:48.605426 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:48.605431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:48.605683 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.605870 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.606166 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":254,"threads_started":1}
I20260924 11:43:48.608525 14699 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35925
I20260924 11:43:48.608733 14699 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:48.609262 14699 heartbeater.cc:511] Master 127.0.97.62:35925 requested a full tablet report, sending...
I20260924 11:43:48.610445 14243 ts_manager.cc:194] Registered new tserver with Master: 7c7b3b23c7274c8b8f1b98bf402afbe3 (127.0.97.3:33125)
I20260924 11:43:48.611202 14243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41039
I20260924 11:43:48.619951 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:43:48.634720 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.623995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.0.97.62:35925)
0924 11:43:48.624123 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.624127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.624639 (+ 512us) server_negotiation.cc:407] Connection header received
0924 11:43:48.624752 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.624755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.624805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.624877 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:48.626097 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.626702 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.627860 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.628281 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.628360 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.630384 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:48.630388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:48.630391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:48.630416 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:48.632784 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:48.633303 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:48.633306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:48.633308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:48.633407 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:48.633706 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:48.633709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:48.633711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:48.633956 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:48.634123 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.634381 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.634525 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:43:48.638144 14243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:43:48.644507 14243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:43:48.665104 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.654348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.0.97.3:33125)
0924 11:43:48.657688 (+ 3340us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.657695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.657722 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:43:48.657799 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.657806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.657996 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.658173 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:48.658828 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.660269 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.663048 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.664038 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.664199 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.664416 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.664637 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.664784 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3204,"thread_start_us":3053,"threads_started":1}
I20260924 11:43:48.665465 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.654058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33125 (local address 127.0.0.1:45144)
0924 11:43:48.654764 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.654810 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.654975 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.658171 (+ 3196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.658175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.658197 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:48.658593 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.658609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.660392 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.660396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.662870 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.662881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.664240 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.664246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.664412 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.664465 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.664610 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.664777 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":146,"thread_start_us":382,"threads_started":1}
I20260924 11:43:48.665887 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.654717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42544)
0924 11:43:48.655561 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.655581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.655681 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.657684 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.657691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.657720 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:48.658027 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.658044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.660271 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.660278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.663153 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.663162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.665072 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.665077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.665268 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.665307 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.665378 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.665487 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":140,"thread_start_us":1048,"threads_started":1}
I20260924 11:43:48.666674 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.655280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45799 (local address 127.0.0.1:54572)
0924 11:43:48.657388 (+ 2108us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.657407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.657532 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.658469 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.658476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.658517 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:48.659555 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.659566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.661139 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.661162 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.663253 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.663266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.665058 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.665065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.666116 (+ 1051us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.666143 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.666227 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.666331 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"mutex_wait_us":1041,"thread_start_us":1801,"threads_started":1}
I20260924 11:43:48.666738 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.655684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.0.97.2:45799)
0924 11:43:48.657987 (+ 2303us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.657993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.658015 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:43:48.658082 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.658086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.658284 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.658473 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:48.659716 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.660998 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.663398 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.664777 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.665250 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.665445 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.666289 (+ 844us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.666440 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2172,"thread_start_us":2055,"threads_started":1}
I20260924 11:43:48.667523 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.655178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.0.97.1:33745)
0924 11:43:48.656378 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.656388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.656420 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:43:48.656536 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.656546 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.656796 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.658195 (+ 1399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:48.658458 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.660104 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.663292 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.664771 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.666836 (+ 2065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.667013 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.667087 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.667250 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":893,"threads_started":1}
I20260924 11:43:48.668898 14634 tablet_service.cc:1467] Processing CreateTablet for tablet 9dedcec5d5204d87b2f05fc2503603d3 (DEFAULT_TABLE table=test-table [id=962927b64df242da82ec565b9168e61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:48.670288 14634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dedcec5d5204d87b2f05fc2503603d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:48.670495 14367 tablet_service.cc:1467] Processing CreateTablet for tablet 9dedcec5d5204d87b2f05fc2503603d3 (DEFAULT_TABLE table=test-table [id=962927b64df242da82ec565b9168e61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:48.671697 14367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dedcec5d5204d87b2f05fc2503603d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:48.672281 14501 tablet_service.cc:1467] Processing CreateTablet for tablet 9dedcec5d5204d87b2f05fc2503603d3 (DEFAULT_TABLE table=test-table [id=962927b64df242da82ec565b9168e61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:48.673281 14501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dedcec5d5204d87b2f05fc2503603d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:48.683173 14717 tablet_bootstrap.cc:492] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Bootstrap starting.
I20260924 11:43:48.684952 14718 tablet_bootstrap.cc:492] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Bootstrap starting.
I20260924 11:43:48.686443 14719 tablet_bootstrap.cc:492] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Bootstrap starting.
I20260924 11:43:48.687508 14717 tablet_bootstrap.cc:654] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:48.688282 14718 tablet_bootstrap.cc:654] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:48.688766 14717 log.cc:824] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:48.689417 14718 log.cc:824] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:48.689746 14719 tablet_bootstrap.cc:654] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:48.690934 14719 log.cc:824] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:48.693423 14719 tablet_bootstrap.cc:492] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: No bootstrap required, opened a new log
I20260924 11:43:48.693660 14719 ts_tablet_manager.cc:1397] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 11:43:48.697716 14717 tablet_bootstrap.cc:492] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: No bootstrap required, opened a new log
I20260924 11:43:48.698037 14717 ts_tablet_manager.cc:1397] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260924 11:43:48.704727 14719 raft_consensus.cc:348] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.705233 14719 raft_consensus.cc:374] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:48.705368 14719 raft_consensus.cc:729] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62055ee072fa40b5ba835375ee3233e2, State: Initialized, Role: FOLLOWER
I20260924 11:43:48.706175 14719 consensus_queue.cc:260] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.708670 14717 raft_consensus.cc:348] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.709055 14717 raft_consensus.cc:374] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:48.709163 14717 raft_consensus.cc:729] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c698cf01febc437bba468e3f996726f1, State: Initialized, Role: FOLLOWER
I20260924 11:43:48.709295 14718 tablet_bootstrap.cc:492] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: No bootstrap required, opened a new log
I20260924 11:43:48.709615 14718 ts_tablet_manager.cc:1397] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.023s
I20260924 11:43:48.709758 14717 consensus_queue.cc:260] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.711028 14432 heartbeater.cc:503] Master 127.0.97.62:35925 was elected leader, sending a full tablet report...
I20260924 11:43:48.711476 14719 ts_tablet_manager.cc:1428] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260924 11:43:48.712798 14566 heartbeater.cc:503] Master 127.0.97.62:35925 was elected leader, sending a full tablet report...
I20260924 11:43:48.713388 14717 ts_tablet_manager.cc:1428] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 11:43:48.718241 14718 raft_consensus.cc:348] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.718538 14718 raft_consensus.cc:374] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:48.718633 14718 raft_consensus.cc:729] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7b3b23c7274c8b8f1b98bf402afbe3, State: Initialized, Role: FOLLOWER
I20260924 11:43:48.719125 14718 consensus_queue.cc:260] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.720089 14699 heartbeater.cc:503] Master 127.0.97.62:35925 was elected leader, sending a full tablet report...
I20260924 11:43:48.720707 14718 ts_tablet_manager.cc:1428] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
W20260924 11:43:48.839645 14700 tablet.cc:2369] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:43:48.891630 14567 tablet.cc:2369] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:48.920019 14725 raft_consensus.cc:482] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:43:48.920329 14725 raft_consensus.cc:504] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.921947 14725 leader_election.cc:288] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799), 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745)
I20260924 11:43:48.927776 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.922306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.97.3:35825)
0924 11:43:48.922511 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.922530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.922683 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.923129 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.923135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.923170 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:48.923391 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.923403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.924421 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.924429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.926069 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.926083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.927079 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.927085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.927273 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.927296 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.927399 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.927535 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 11:43:48.927776 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.922309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35825 (local address 127.0.97.1:33745)
0924 11:43:48.922515 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.922524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.922615 (+ 91us) server_negotiation.cc:407] Connection header received
0924 11:43:48.922868 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.922873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.922970 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.923078 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:48.923522 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.924288 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.926207 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.926914 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.927067 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.927155 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.927423 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.927543 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:43:48.928607 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.922411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34245 (local address 127.0.97.2:45799)
0924 11:43:48.922607 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.922612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.923576 (+ 964us) server_negotiation.cc:407] Connection header received
0924 11:43:48.923759 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.923764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.923842 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.923933 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:48.924532 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.925183 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.926995 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.927721 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.927870 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.927961 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.928257 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.928375 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:43:48.928665 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.922737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45799 (local address 127.0.97.3:34245)
0924 11:43:48.923481 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:48.923499 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:48.923609 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:48.923945 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:48.923949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:48.923971 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:48.924333 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.924361 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.925323 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.925328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.926842 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:48.926853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.927905 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.927911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:48.928075 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.928095 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:43:48.928230 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 11:43:48.928343 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":384,"threads_started":1}
I20260924 11:43:48.929010 14387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dedcec5d5204d87b2f05fc2503603d3" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62055ee072fa40b5ba835375ee3233e2" is_pre_election: true
I20260924 11:43:48.929551 14387 raft_consensus.cc:2454] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 0.
I20260924 11:43:48.929620 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dedcec5d5204d87b2f05fc2503603d3" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c698cf01febc437bba468e3f996726f1" is_pre_election: true
I20260924 11:43:48.930122 14521 raft_consensus.cc:2454] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 0.
I20260924 11:43:48.930245 14589 leader_election.cc:302] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62055ee072fa40b5ba835375ee3233e2, 7c7b3b23c7274c8b8f1b98bf402afbe3; no voters:
I20260924 11:43:48.930626 14725 raft_consensus.cc:2781] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:43:48.930787 14725 raft_consensus.cc:482] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:43:48.930929 14725 raft_consensus.cc:3037] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:48.933091 14725 raft_consensus.cc:504] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.933636 14725 leader_election.cc:288] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 election: Requested vote from peers c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799), 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745)
I20260924 11:43:48.934046 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dedcec5d5204d87b2f05fc2503603d3" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c698cf01febc437bba468e3f996726f1"
I20260924 11:43:48.934079 14387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dedcec5d5204d87b2f05fc2503603d3" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62055ee072fa40b5ba835375ee3233e2"
I20260924 11:43:48.934226 14521 raft_consensus.cc:3037] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:48.934255 14387 raft_consensus.cc:3037] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:48.936218 14521 raft_consensus.cc:2454] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 1.
I20260924 11:43:48.936285 14387 raft_consensus.cc:2454] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 1.
I20260924 11:43:48.936594 14590 leader_election.cc:302] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7b3b23c7274c8b8f1b98bf402afbe3, c698cf01febc437bba468e3f996726f1; no voters:
I20260924 11:43:48.936857 14725 raft_consensus.cc:2781] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:48.937306 14725 raft_consensus.cc:686] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7b3b23c7274c8b8f1b98bf402afbe3, State: Running, Role: LEADER
I20260924 11:43:48.937736 14725 consensus_queue.cc:237] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } }
I20260924 11:43:48.941826 14242 catalog_manager.cc:5495] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c7b3b23c7274c8b8f1b98bf402afbe3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } health_report { overall_health: UNKNOWN } } }
W20260924 11:43:48.943333 14433 tablet.cc:2369] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:48.956782 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.952393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.0.97.3:33125)
0924 11:43:48.952553 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.952557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:48.952653 (+ 96us) server_negotiation.cc:407] Connection header received
0924 11:43:48.952770 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:48.952773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:48.952827 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:48.952920 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:43:48.953215 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.953859 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.955211 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:48.955533 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:48.955608 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:48.955979 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:43:48.956424 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:43:48.956521 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:48.956579 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 11:43:48.956629 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:43:48.963670 14521 raft_consensus.cc:1261] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Refusing update from remote peer 7c7b3b23c7274c8b8f1b98bf402afbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:48.963670 14387 raft_consensus.cc:1261] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Refusing update from remote peer 7c7b3b23c7274c8b8f1b98bf402afbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:48.964661 14725 consensus_queue.cc:1035] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:48.964982 14727 consensus_queue.cc:1035] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:48.987118 14242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:43:48.987386 14242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:43:48.990540 14242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Sending DeleteTablet for 3 replicas of tablet 9dedcec5d5204d87b2f05fc2503603d3
I20260924 11:43:48.991483 14634 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dedcec5d5204d87b2f05fc2503603d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45144
I20260924 11:43:48.991514 14501 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dedcec5d5204d87b2f05fc2503603d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54572
I20260924 11:43:48.991528 14367 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dedcec5d5204d87b2f05fc2503603d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42544
I20260924 11:43:48.992527 14741 tablet_replica.cc:330] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: stopping tablet replica
I20260924 11:43:48.992799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14304
I20260924 11:43:48.992913 14740 tablet_replica.cc:330] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: stopping tablet replica
I20260924 11:43:48.993322 14740 raft_consensus.cc:2229] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:43:48.993842 14741 raft_consensus.cc:2229] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:43:48.994464 14740 raft_consensus.cc:2258] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:48.995209 14741 raft_consensus.cc:2258] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:48.997601 14741 ts_tablet_manager.cc:1905] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:49.007589 14741 ts_tablet_manager.cc:1918] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:49.009217 14740 ts_tablet_manager.cc:1905] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:43:49 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:43:49.014277 14742 tablet_replica.cc:330] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: stopping tablet replica
I20260924 11:43:49.014652 14742 raft_consensus.cc:2229] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:43:49.015112 14742 raft_consensus.cc:2258] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:49.018127 14742 ts_tablet_manager.cc:1905] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:49.021488 14741 log.cc:1196] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/9dedcec5d5204d87b2f05fc2503603d3
I20260924 11:43:49.022105 14741 ts_tablet_manager.cc:1939] T 9dedcec5d5204d87b2f05fc2503603d3 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Deleting consensus metadata
I20260924 11:43:49.023468 14740 ts_tablet_manager.cc:1918] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:49.023963 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:48.993813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.0.97.1:33745)
0924 11:43:48.993978 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:48.993985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:49.003452 (+ 9467us) server_negotiation.cc:407] Connection header received
0924 11:43:49.003831 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:49.003837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:49.003911 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:49.003993 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:49.005834 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:49.006877 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:49.008870 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:49.009480 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:49.009567 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:49.016480 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:49.016492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:49.016507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:49.016544 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:49.020457 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:49.021340 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:49.021348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:49.021352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:49.021512 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:49.022017 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:49.022022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:49.022026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:49.022477 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:49.022780 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:49.023340 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 11:43:49.023570 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:43:49.025223 14740 log.cc:1196] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/9dedcec5d5204d87b2f05fc2503603d3
I20260924 11:43:49.025350 14427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:49.025568 14230 catalog_manager.cc:4841] TS 7c7b3b23c7274c8b8f1b98bf402afbe3 (127.0.97.3:33125): tablet 9dedcec5d5204d87b2f05fc2503603d3 (table test-table [id=962927b64df242da82ec565b9168e61e]) successfully deleted
I20260924 11:43:49.025991 14742 ts_tablet_manager.cc:1918] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:49.026130 14742 log.cc:1196] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/9dedcec5d5204d87b2f05fc2503603d3
I20260924 11:43:49.026603 14742 ts_tablet_manager.cc:1939] T 9dedcec5d5204d87b2f05fc2503603d3 P c698cf01febc437bba468e3f996726f1: Deleting consensus metadata
I20260924 11:43:49.028129 14230 catalog_manager.cc:4841] TS c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799): tablet 9dedcec5d5204d87b2f05fc2503603d3 (table test-table [id=962927b64df242da82ec565b9168e61e]) successfully deleted
I20260924 11:43:49.125339 14740 ts_tablet_manager.cc:1939] T 9dedcec5d5204d87b2f05fc2503603d3 P 62055ee072fa40b5ba835375ee3233e2: Deleting consensus metadata
I20260924 11:43:49.126042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14304
W20260924 11:43:49.147944 14229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33745 (error 108)
I20260924 11:43:49.148355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14438
I20260924 11:43:49.152602 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.152090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42566)
0924 11:43:49.152335 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.152439 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260924 11:43:49.152848 14229 catalog_manager.cc:4584] TS 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dedcec5d5204d87b2f05fc2503603d3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Sep 24 11:43:49 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:43:49.166805 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.149475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.97.2:45799)
0924 11:43:49.149648 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:49.149654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:49.149677 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:43:49.149736 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:49.149741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:49.149834 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:49.149952 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:49.151740 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:49.152880 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:49.154843 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:49.155480 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:49.155594 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:49.160946 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:49.160958 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:49.160975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:49.161017 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:49.164048 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:49.164715 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:49.164723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:49.164733 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:49.164848 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:49.165221 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:49.165226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:49.165228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:49.165549 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:49.165818 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:49.166318 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 11:43:49.166529 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:43:49.167464 14561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:49.206410 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.205946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42578)
0924 11:43:49.206162 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.206262 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 11:43:49.207475 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.207109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42584)
0924 11:43:49.207259 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.207327 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:43:49.252123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14438
I20260924 11:43:49.267851 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.267452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42596)
0924 11:43:49.267630 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.267704 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 11:43:49.268792 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.268494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42598)
0924 11:43:49.268626 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.268698 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 11:43:49.274451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14571
Sep 24 11:43:49 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:43:49.289247 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.275151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.0.97.3:33125)
0924 11:43:49.275341 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:49.275346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:49.275365 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:43:49.275520 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:49.275541 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:49.275620 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:49.275719 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:49.277008 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:49.277813 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:49.278878 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:49.279270 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:49.279388 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:49.283701 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:49.283715 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:49.283732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:49.283775 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:49.286625 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:49.287339 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:49.287344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:49.287349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:49.287489 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:49.287847 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:49.287849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:49.287851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:49.288123 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:49.288509 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:49.288817 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 11:43:49.288947 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:43:49.290012 14694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:49.349710 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.349171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42600)
0924 11:43:49.349443 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.349559 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 11:43:49.350685 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:49.350328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42612)
0924 11:43:49.350515 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:49.350597 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 11:43:49.403285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14571
I20260924 11:43:49.422003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14211
I20260924 11:43:49.423100 14273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:49.526147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14211
I20260924 11:43:49.546427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35925
--webserver_interface=127.0.97.62
--webserver_port=44993
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:49.840235 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:49.840530 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:49.840683 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:49.847695 14749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:49.847822 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:49.847894 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:49.847970 14749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:49.848043 14749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:49.856560 14749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:49.857587 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:49.858695 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:49.879515 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:49.879673 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:49.879618 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:49.880446 14749 server_base.cc:1034] running on GCE node
I20260924 11:43:49.881173 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:49.882530 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:49.883715 14749 hybrid_clock.cc:648] HybridClock initialized: now 1790250229883670 us; error 34 us; skew 500 ppm
I20260924 11:43:49.884063 14749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:49 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:49.887343 14749 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:49.888826 14749 webserver.cc:460] Webserver started at http://127.0.97.62:44993/ using document root <none> and password file <none>
I20260924 11:43:49.889508 14749 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:49.889657 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:49.894876 14749 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 11:43:49.897517 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:49.898705 14749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:43:49.898878 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "2f30d11e5e45413db6dda133ac2004b8"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "957e4840b73ff46f3b9f260d90265cf5"
server_key_iv: "390b237bac74d28dc8f0fa762eb18d65"
server_key_version: "encryptionkey@0"
I20260924 11:43:49.899893 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:49.920049 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:49.924013 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:49.924304 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:49.942153 14749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35925
I20260924 11:43:49.942183 14816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35925 every 8 connection(s)
I20260924 11:43:49.943467 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:49.946113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14749
I20260924 11:43:49.946807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33745
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35925
--webserver_port=44253
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:43:49.949569 14817 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:43:49.952608 14817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap starting.
I20260924 11:43:49.979826 14817 log.cc:824] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:49.993273 14817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:43:49.993827 14817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap complete.
I20260924 11:43:50.007197 14817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:50.008628 14817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f30d11e5e45413db6dda133ac2004b8, State: Initialized, Role: FOLLOWER
I20260924 11:43:50.009442 14817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:50.009757 14817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:43:50.009903 14817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:43:50.010133 14817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:43:50.015709 14817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:50.016345 14817 leader_election.cc:302] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f30d11e5e45413db6dda133ac2004b8; no voters:
I20260924 11:43:50.016965 14817 leader_election.cc:288] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:43:50.017102 14821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:43:50.018260 14821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 LEADER]: Becoming Leader. State: Replica: 2f30d11e5e45413db6dda133ac2004b8, State: Running, Role: LEADER
I20260924 11:43:50.018720 14821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:50.019009 14817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:50.021060 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f30d11e5e45413db6dda133ac2004b8. Latest consensus state: current_term: 2 leader_uuid: "2f30d11e5e45413db6dda133ac2004b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } } }
I20260924 11:43:50.021449 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:50.021950 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f30d11e5e45413db6dda133ac2004b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } } }
I20260924 11:43:50.022233 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:50.032841 14827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:50.045908 14827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:50.047735 14827 catalog_manager.cc:1259] Loaded cluster ID: 40f269c04f714abd949f3dac8c9b399f
I20260924 11:43:50.047837 14827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:43:50.051692 14827 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 11:43:50.081367 14827 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 11:43:50.099468 14827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:43:50.100791 14827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Loaded TSK: 0
I20260924 11:43:50.102028 14827 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:43:50.314025 14819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:50.314254 14819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:50.314388 14819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:50.321516 14819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:50.321604 14819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:50.321801 14819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:43:50.330533 14819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44253
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:50.331593 14819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:50.332804 14819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:50.344451 14844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:50.345048 14845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:50.345112 14847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:50.345697 14819 server_base.cc:1034] running on GCE node
I20260924 11:43:50.346261 14819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:50.347126 14819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:50.348335 14819 hybrid_clock.cc:648] HybridClock initialized: now 1790250230348290 us; error 41 us; skew 500 ppm
I20260924 11:43:50.348649 14819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:50 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:50.351349 14819 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:43:50.352785 14819 webserver.cc:460] Webserver started at http://127.0.97.1:44253/ using document root <none> and password file <none>
I20260924 11:43:50.353348 14819 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:50.353489 14819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:50.358318 14819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:43:50.360845 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:50.362164 14819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:43:50.362310 14819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "62055ee072fa40b5ba835375ee3233e2"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "b20f0016f3baa7842166b78d99660885"
server_key_iv: "936cac53dff84e69af2a556fa4060e66"
server_key_version: "encryptionkey@0"
I20260924 11:43:50.363173 14819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:50.381721 14819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:50.385736 14819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:50.386000 14819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:50.386946 14819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:50.388258 14819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:50.388341 14819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:50.388465 14819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:50.388536 14819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:50.428120 14819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33745
I20260924 11:43:50.428232 14966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33745 every 8 connection(s)
I20260924 11:43:50.429596 14819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:43:50.430678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14819
I20260924 11:43:50.431354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45799
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35925
--webserver_port=45043
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:43:50 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:50.452854 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:50.432624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:55449)
0924 11:43:50.433492 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:50.433538 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:50.435169 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:50.436622 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:50.436630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:50.437161 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:50.437694 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:50.437709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.439243 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.439250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:50.441141 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:50.441158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.442165 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.442174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:50.442487 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:50.443118 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:50.443150 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:50.445712 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:50.448867 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:50.448874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:50.448886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:50.449288 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:50.449726 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:50.449731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:50.449735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:50.449898 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:50.450579 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:50.450604 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:50.450912 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 11:43:50.451141 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 11:43:50.451628 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":374,"threads_started":1}
I20260924 11:43:50.452854 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:50.432404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55449 (local address 127.0.97.62:35925)
0924 11:43:50.433044 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:50.433060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:50.433679 (+ 619us) server_negotiation.cc:407] Connection header received
0924 11:43:50.435696 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:50.435716 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:50.436124 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:50.436533 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:50.437911 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.439096 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.441292 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.441935 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.442071 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:50.445964 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:50.445971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:50.445981 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:50.446012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:50.448719 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:50.449444 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:50.449463 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:50.449468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:50.449614 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:50.450013 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:50.450022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:50.450024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:50.450435 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:50.450663 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:50.451334 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 11:43:50.451705 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260924 11:43:50.455372 14967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35925
I20260924 11:43:50.455655 14967 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:50.456346 14967 heartbeater.cc:511] Master 127.0.97.62:35925 requested a full tablet report, sending...
I20260924 11:43:50.458209 14782 ts_manager.cc:194] Registered new tserver with Master: 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745)
I20260924 11:43:50.459627 14782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55449
W20260924 11:43:50.733279 14971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:50.733575 14971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:50.733728 14971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:50.740319 14971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:50.740447 14971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:50.740595 14971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:43:50.749059 14971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45043
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:50.750066 14971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:50.751260 14971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:50.763346 14979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:50.763305 14976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:50.763366 14977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:50.764024 14971 server_base.cc:1034] running on GCE node
I20260924 11:43:50.764590 14971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:50.765488 14971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:50.766714 14971 hybrid_clock.cc:648] HybridClock initialized: now 1790250230766660 us; error 46 us; skew 500 ppm
I20260924 11:43:50.766999 14971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:50 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:50.769709 14971 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:43:50.770970 14971 webserver.cc:460] Webserver started at http://127.0.97.2:45043/ using document root <none> and password file <none>
I20260924 11:43:50.771548 14971 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:50.771709 14971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:50.776443 14971 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:43:50.778633 14986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:50.779740 14971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:43:50.779888 14971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "c698cf01febc437bba468e3f996726f1"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "fd52279d8d26d22b9c66e4a3d8d3aebf"
server_key_iv: "1ecbac803d7d99a24924ef4cafc952c6"
server_key_version: "encryptionkey@0"
I20260924 11:43:50.780755 14971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:50.805243 14971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:50.808458 14971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:50.808691 14971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:50.809468 14971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:50.810652 14971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:50.810783 14971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:50.810889 14971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:50.810972 14971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:50.841585 14971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45799
I20260924 11:43:50.841661 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45799 every 8 connection(s)
I20260924 11:43:50.842944 14971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:43:50.852967 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 14971
I20260924 11:43:50.853570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33125
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35925
--webserver_port=38175
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:43:50 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:50.860486 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:50.845189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52465 (local address 127.0.97.62:35925)
0924 11:43:50.845365 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:50.845369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:50.846234 (+ 865us) server_negotiation.cc:407] Connection header received
0924 11:43:50.847784 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:50.847789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:50.847870 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:50.847971 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:50.849276 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.849934 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.851299 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.851719 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.851844 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:50.855165 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:50.855170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:50.855173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:50.855202 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:50.857610 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:50.858308 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:50.858314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:50.858317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:50.858460 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:50.858843 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:50.858848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:50.858851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:50.859126 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:50.859275 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:50.859893 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 11:43:50.860127 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:43:50.861323 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:50.845519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:52465)
0924 11:43:50.846096 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:50.846127 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:50.847551 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:50.848224 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:50.848231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:50.848631 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:50.849114 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:50.849129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.850082 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.850085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:50.851110 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:50.851126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:50.851937 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:50.851945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:50.852211 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:50.852830 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:50.852848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:50.854962 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:50.857745 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:50.857753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:50.857765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:50.858184 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:50.858561 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:50.858567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:50.858570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:50.858730 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:50.859265 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:50.859272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:50.859543 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 11:43:50.859802 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:43:50.860294 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":234,"threads_started":1}
I20260924 11:43:50.862675 15099 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35925
I20260924 11:43:50.862938 15099 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:50.863559 15099 heartbeater.cc:511] Master 127.0.97.62:35925 requested a full tablet report, sending...
I20260924 11:43:50.864974 14782 ts_manager.cc:194] Registered new tserver with Master: c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799)
I20260924 11:43:50.866144 14782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52465
W20260924 11:43:51.157411 15102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:51.157655 15102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:51.157770 15102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:51.164649 15102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:51.164758 15102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:51.164935 15102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:43:51.173454 15102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38175
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:51.174765 15102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:51.175936 15102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:51.187928 15107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:51.187908 15108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:51.188027 15110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:51.188853 15102 server_base.cc:1034] running on GCE node
I20260924 11:43:51.189359 15102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:51.190163 15102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:51.191372 15102 hybrid_clock.cc:648] HybridClock initialized: now 1790250231191318 us; error 43 us; skew 500 ppm
I20260924 11:43:51.191651 15102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:51 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:51.194554 15102 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:43:51.195706 15102 webserver.cc:460] Webserver started at http://127.0.97.3:38175/ using document root <none> and password file <none>
I20260924 11:43:51.196265 15102 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:51.196367 15102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:51.200922 15102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:43:51.203272 15117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:51.204696 15102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:43:51.204810 15102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "a9aa9bc6555a461cdcaa2d623139758a"
server_key_iv: "6f0253b4598f7e2e7c40f84c30da3140"
server_key_version: "encryptionkey@0"
I20260924 11:43:51.205626 15102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:51.238566 15102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:51.242473 15102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:51.242684 15102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:51.243432 15102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:51.244577 15102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:51.244658 15102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:51.244730 15102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:51.244784 15102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:51.273985 15102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33125
I20260924 11:43:51.274067 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33125 every 8 connection(s)
I20260924 11:43:51.275197 15102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:43:51.284705 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15102
Sep 24 11:43:51 dist-test-slave-qhrh krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:43:51.294124 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.277392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54457 (local address 127.0.97.62:35925)
0924 11:43:51.277559 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.277563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.278619 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 11:43:51.280213 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.280218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.280297 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.280412 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:51.281679 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.282365 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.283777 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.284380 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.284511 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.288883 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:51.288888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:51.288891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:51.288919 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:51.291445 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.292093 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.292098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.292102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.292263 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.292628 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.292649 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.292652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.292870 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:51.293020 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.293556 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.293767 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:43:51.294883 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.277776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:54457)
0924 11:43:51.278491 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.278525 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:51.279932 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:51.280620 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:51.280627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:51.281040 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:43:51.281526 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.281541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.282511 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.282521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.283625 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.283656 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.284606 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.284614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.284859 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.286067 (+ 1208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:43:51.286100 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:43:51.288660 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:43:51.291564 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:51.291571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:51.291583 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:51.291984 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:51.292359 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:43:51.292364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:43:51.292367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:43:51.292522 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:43:51.293009 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:43:51.293015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:43:51.293261 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 11:43:51.293483 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 11:43:51.293894 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":325,"threads_started":1}
I20260924 11:43:51.296236 15230 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35925
I20260924 11:43:51.296643 15230 heartbeater.cc:464] Registering TS with master...
I20260924 11:43:51.297330 15230 heartbeater.cc:511] Master 127.0.97.62:35925 requested a full tablet report, sending...
I20260924 11:43:51.298684 14782 ts_manager.cc:194] Registered new tserver with Master: 7c7b3b23c7274c8b8f1b98bf402afbe3 (127.0.97.3:33125)
I20260924 11:43:51.298959 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.285617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.0.97.62:35925)
0924 11:43:51.286079 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.286084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.286330 (+ 246us) server_negotiation.cc:407] Connection header received
0924 11:43:51.286478 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.286481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.286525 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.286599 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:51.288486 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.289394 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.290781 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.291193 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.291317 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.293558 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:51.293563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:51.293569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:51.293598 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:51.296479 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.297099 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.297106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.297109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.297222 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.297616 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.297621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.297624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.297849 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:51.298078 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.298368 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.298538 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260924 11:43:51.299579 14782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54457
I20260924 11:43:51.302251 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:43:51.308055 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.303041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.0.97.62:35925)
0924 11:43:51.303176 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.303180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.303639 (+ 459us) server_negotiation.cc:407] Connection header received
0924 11:43:51.303819 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.303823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.303977 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.304100 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:43:51.304513 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.305146 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.306430 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.306807 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.306898 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.307259 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:43:51.307614 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:43:51.307703 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.307783 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.307831 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:43:51.308832 14782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:43:51.315259 14782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:43:51.348274 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.325457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42638)
0924 11:43:51.332558 (+ 7101us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.332600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:51.332756 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:51.341793 (+ 9037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:51.341799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:51.341827 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:51.342243 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.342254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.343569 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.343575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.345788 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.345799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.347437 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.347443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.347615 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.347667 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 11:43:51.347789 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 11:43:51.347926 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6863,"thread_start_us":15248,"threads_started":1}
I20260924 11:43:51.349072 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.325672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.97.1:33745)
0924 11:43:51.341295 (+ 15623us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.341303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.341334 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:43:51.341409 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.341414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.341559 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.342454 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:51.342648 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.343449 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.345931 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.347210 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.347325 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.347515 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.348607 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.348764 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15494,"thread_start_us":15354,"threads_started":1}
I20260924 11:43:51.351720 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.326935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33125 (local address 127.0.0.1:45166)
0924 11:43:51.344564 (+ 17629us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.344581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:51.344694 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:51.345496 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:51.345501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:51.345523 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:51.345991 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.346001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.347919 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.347923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.349863 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.349877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.350984 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.350990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.351153 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.351185 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:43:51.351287 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:43:51.351400 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17505,"mutex_wait_us":43,"thread_start_us":17274,"threads_started":1}
I20260924 11:43:51.351732 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.327474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.0.97.3:33125)
0924 11:43:51.344013 (+ 16539us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.344023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.344828 (+ 805us) server_negotiation.cc:407] Connection header received
0924 11:43:51.344925 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.344933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.345145 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.345286 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:51.346154 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.347788 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.350034 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.350782 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.350937 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.351131 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.351320 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.351447 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16358,"thread_start_us":16232,"threads_started":1}
I20260924 11:43:51.351954 14902 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3ede37f118405fa0acacc0b437b7a4 (DEFAULT_TABLE table=test-table [id=1c90593236404a22abe35edc26077fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:51.352957 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.326437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.0.97.2:45799)
0924 11:43:51.344778 (+ 18341us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.344784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.344808 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:43:51.344876 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.344880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.345042 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.345220 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:51.345842 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.346823 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.348897 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.349841 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.349969 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.350193 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.352486 (+ 2293us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.352647 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18205,"thread_start_us":18078,"threads_started":1}
I20260924 11:43:51.353400 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3ede37f118405fa0acacc0b437b7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:51.353647 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.325808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45799 (local address 127.0.0.1:54596)
0924 11:43:51.340475 (+ 14667us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.340495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:51.340616 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:51.345210 (+ 4594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:51.345214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:51.345234 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:51.345646 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.345659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.346954 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.346959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.348770 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.348781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.352040 (+ 3259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.352063 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.352280 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.352306 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:43:51.352421 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 11:43:51.352542 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14530,"mutex_wait_us":2,"thread_start_us":415,"threads_started":1}
I20260924 11:43:51.355217 15165 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3ede37f118405fa0acacc0b437b7a4 (DEFAULT_TABLE table=test-table [id=1c90593236404a22abe35edc26077fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:51.356383 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3ede37f118405fa0acacc0b437b7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:51.356768 15034 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3ede37f118405fa0acacc0b437b7a4 (DEFAULT_TABLE table=test-table [id=1c90593236404a22abe35edc26077fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:43:51.357879 15034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3ede37f118405fa0acacc0b437b7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:43:51.367609 15246 tablet_bootstrap.cc:492] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Bootstrap starting.
I20260924 11:43:51.369835 15247 tablet_bootstrap.cc:492] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Bootstrap starting.
I20260924 11:43:51.371168 15248 tablet_bootstrap.cc:492] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Bootstrap starting.
I20260924 11:43:51.371433 15246 tablet_bootstrap.cc:654] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:51.372731 15246 log.cc:824] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:51.373342 15247 tablet_bootstrap.cc:654] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:51.373701 15248 tablet_bootstrap.cc:654] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:43:51.374621 15247 log.cc:824] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:51.374845 15248 log.cc:824] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:51.376402 15246 tablet_bootstrap.cc:492] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: No bootstrap required, opened a new log
I20260924 11:43:51.376645 15246 ts_tablet_manager.cc:1397] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:43:51.377453 15247 tablet_bootstrap.cc:492] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: No bootstrap required, opened a new log
I20260924 11:43:51.377727 15247 ts_tablet_manager.cc:1397] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260924 11:43:51.378211 15248 tablet_bootstrap.cc:492] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: No bootstrap required, opened a new log
I20260924 11:43:51.378441 15248 ts_tablet_manager.cc:1397] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 11:43:51.385335 15247 raft_consensus.cc:348] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.385680 15247 raft_consensus.cc:374] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:51.385779 15247 raft_consensus.cc:729] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7b3b23c7274c8b8f1b98bf402afbe3, State: Initialized, Role: FOLLOWER
I20260924 11:43:51.386415 15247 consensus_queue.cc:260] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.387348 15246 raft_consensus.cc:348] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.387704 15246 raft_consensus.cc:374] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:51.387797 15246 raft_consensus.cc:729] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62055ee072fa40b5ba835375ee3233e2, State: Initialized, Role: FOLLOWER
I20260924 11:43:51.387795 15248 raft_consensus.cc:348] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.388108 15248 raft_consensus.cc:374] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:43:51.388161 15230 heartbeater.cc:503] Master 127.0.97.62:35925 was elected leader, sending a full tablet report...
I20260924 11:43:51.388211 15248 raft_consensus.cc:729] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c698cf01febc437bba468e3f996726f1, State: Initialized, Role: FOLLOWER
I20260924 11:43:51.388304 15246 consensus_queue.cc:260] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.388648 15247 ts_tablet_manager.cc:1428] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260924 11:43:51.388814 15248 consensus_queue.cc:260] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.390275 15099 heartbeater.cc:503] Master 127.0.97.62:35925 was elected leader, sending a full tablet report...
I20260924 11:43:51.390934 15248 ts_tablet_manager.cc:1428] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 11:43:51.391707 14967 heartbeater.cc:503] Master 127.0.97.62:35925 was elected leader, sending a full tablet report...
I20260924 11:43:51.392246 15246 ts_tablet_manager.cc:1428] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260924 11:43:51.427973 15252 raft_consensus.cc:482] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:43:51.428251 15252 raft_consensus.cc:504] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.430056 15252 leader_election.cc:288] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745), c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799)
W20260924 11:43:51.431710 14968 tablet.cc:2369] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:43:51.436566 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.430442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46319 (local address 127.0.97.2:45799)
0924 11:43:51.430626 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.430633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.430802 (+ 169us) server_negotiation.cc:407] Connection header received
0924 11:43:51.431067 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.431073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.431147 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.431248 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:51.431664 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.432629 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.434947 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.435707 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.435839 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.435940 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.436213 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.436344 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:43:51.436602 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.430405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45799 (local address 127.0.97.3:46319)
0924 11:43:51.430702 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.430725 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:51.430863 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:51.431267 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:51.431272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:51.431295 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:51.431497 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.431509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.432799 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.432805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.434788 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.434802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.435847 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.435853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.436038 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.436060 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:43:51.436181 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 11:43:51.436327 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":25}
I20260924 11:43:51.437625 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.430582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49307 (local address 127.0.97.1:33745)
0924 11:43:51.430733 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.430737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.431327 (+ 590us) server_negotiation.cc:407] Connection header received
0924 11:43:51.431608 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.431613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.431697 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.431757 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:43:51.432556 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.433311 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.435016 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.436308 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.436474 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.436553 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.437268 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.437395 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:43:51.437474 15054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ede37f118405fa0acacc0b437b7a4" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c698cf01febc437bba468e3f996726f1" is_pre_election: true
I20260924 11:43:51.437646 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.430537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.97.3:49307)
0924 11:43:51.431239 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.431255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:43:51.431460 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:43:51.432005 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:43:51.432009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:43:51.432033 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:43:51.432383 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.432392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.433458 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.433463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.434806 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:43:51.434817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.436942 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.436947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:43:51.437107 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.437145 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:43:51.437225 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 11:43:51.437333 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":1,"thread_start_us":394,"threads_started":1}
I20260924 11:43:51.437922 15054 raft_consensus.cc:2454] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 0.
I20260924 11:43:51.438619 15121 leader_election.cc:302] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7b3b23c7274c8b8f1b98bf402afbe3, c698cf01febc437bba468e3f996726f1; no voters:
I20260924 11:43:51.438692 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ede37f118405fa0acacc0b437b7a4" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62055ee072fa40b5ba835375ee3233e2" is_pre_election: true
I20260924 11:43:51.438951 15252 raft_consensus.cc:2781] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:43:51.439082 15252 raft_consensus.cc:482] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:43:51.439213 15252 raft_consensus.cc:3037] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:51.439240 14922 raft_consensus.cc:2454] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 0.
I20260924 11:43:51.442269 15252 raft_consensus.cc:504] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.442833 15252 leader_election.cc:288] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 election: Requested vote from peers 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745), c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799)
I20260924 11:43:51.443192 15054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ede37f118405fa0acacc0b437b7a4" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c698cf01febc437bba468e3f996726f1"
I20260924 11:43:51.443216 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ede37f118405fa0acacc0b437b7a4" candidate_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62055ee072fa40b5ba835375ee3233e2"
I20260924 11:43:51.443346 15054 raft_consensus.cc:3037] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:51.443399 14922 raft_consensus.cc:3037] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:43:51.445859 14922 raft_consensus.cc:2454] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 1.
I20260924 11:43:51.446075 15054 raft_consensus.cc:2454] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7b3b23c7274c8b8f1b98bf402afbe3 in term 1.
I20260924 11:43:51.446187 15120 leader_election.cc:302] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62055ee072fa40b5ba835375ee3233e2, 7c7b3b23c7274c8b8f1b98bf402afbe3; no voters:
I20260924 11:43:51.446445 15252 raft_consensus.cc:2781] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:43:51.446810 15252 raft_consensus.cc:686] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7b3b23c7274c8b8f1b98bf402afbe3, State: Running, Role: LEADER
I20260924 11:43:51.447213 15252 consensus_queue.cc:237] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } }
I20260924 11:43:51.450773 14780 catalog_manager.cc:5495] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c7b3b23c7274c8b8f1b98bf402afbe3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33125 } health_report { overall_health: HEALTHY } } }
I20260924 11:43:51.485832 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.480549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.0.97.3:33125)
0924 11:43:51.480716 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.480721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.480813 (+ 92us) server_negotiation.cc:407] Connection header received
0924 11:43:51.481046 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.481051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.481156 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.481296 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:43:51.481665 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.482378 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.483784 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.484279 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.484440 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.484920 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:43:51.485339 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:43:51.485491 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.485544 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.485610 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:43:51.492280 15054 raft_consensus.cc:1261] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Refusing update from remote peer 7c7b3b23c7274c8b8f1b98bf402afbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:51.492362 14922 raft_consensus.cc:1261] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Refusing update from remote peer 7c7b3b23c7274c8b8f1b98bf402afbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:43:51.493120 15256 consensus_queue.cc:1035] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c698cf01febc437bba468e3f996726f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:51.493374 15252 consensus_queue.cc:1035] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62055ee072fa40b5ba835375ee3233e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:43:51.518702 14780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:43:51.518939 14780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:43:51.522114 14780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Sending DeleteTablet for 3 replicas of tablet 8a3ede37f118405fa0acacc0b437b7a4
I20260924 11:43:51.523170 14902 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3ede37f118405fa0acacc0b437b7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42638
I20260924 11:43:51.523277 15034 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3ede37f118405fa0acacc0b437b7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54596
I20260924 11:43:51.523489 15165 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3ede37f118405fa0acacc0b437b7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:43:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45166
I20260924 11:43:51.524606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14819
I20260924 11:43:51.534454 15270 tablet_replica.cc:330] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: stopping tablet replica
I20260924 11:43:51.535022 15270 raft_consensus.cc:2229] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:43:51.535670 15270 raft_consensus.cc:2258] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:51.536365 15268 tablet_replica.cc:330] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: stopping tablet replica
I20260924 11:43:51.536701 15268 raft_consensus.cc:2229] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:43:51.537146 15268 raft_consensus.cc:2258] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:51.538944 15270 ts_tablet_manager.cc:1905] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:51.538937 15268 ts_tablet_manager.cc:1905] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:51.541416 15269 tablet_replica.cc:330] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: stopping tablet replica
I20260924 11:43:51.541851 15269 raft_consensus.cc:2229] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:43:51.542344 15269 raft_consensus.cc:2258] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:43:51.542783 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.525436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.97.1:33745)
0924 11:43:51.525610 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.525619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.525642 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:43:51.525793 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.525800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.525910 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.526025 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:51.527661 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.529107 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.530974 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.531833 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.531960 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.535228 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:51.535251 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:51.535263 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:51.535298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:51.539661 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.540356 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.540363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.540366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.540495 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.540835 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.540839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.540841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.541401 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:51.541767 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.542130 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.542362 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:43:51.543830 14962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:51.545802 15269 ts_tablet_manager.cc:1905] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:51.551434 15269 ts_tablet_manager.cc:1918] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:51.551575 15269 log.cc:1196] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/8a3ede37f118405fa0acacc0b437b7a4
I20260924 11:43:51.551842 15270 ts_tablet_manager.cc:1918] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:51.551990 15270 log.cc:1196] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/8a3ede37f118405fa0acacc0b437b7a4
I20260924 11:43:51.552065 15269 ts_tablet_manager.cc:1939] T 8a3ede37f118405fa0acacc0b437b7a4 P c698cf01febc437bba468e3f996726f1: Deleting consensus metadata
I20260924 11:43:51.552515 15270 ts_tablet_manager.cc:1939] T 8a3ede37f118405fa0acacc0b437b7a4 P 7c7b3b23c7274c8b8f1b98bf402afbe3: Deleting consensus metadata
I20260924 11:43:51.553725 14769 catalog_manager.cc:4841] TS c698cf01febc437bba468e3f996726f1 (127.0.97.2:45799): tablet 8a3ede37f118405fa0acacc0b437b7a4 (table test-table [id=1c90593236404a22abe35edc26077fb1]) successfully deleted
I20260924 11:43:51.554193 14769 catalog_manager.cc:4841] TS 7c7b3b23c7274c8b8f1b98bf402afbe3 (127.0.97.3:33125): tablet 8a3ede37f118405fa0acacc0b437b7a4 (table test-table [id=1c90593236404a22abe35edc26077fb1]) successfully deleted
I20260924 11:43:51.630748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14819
W20260924 11:43:51.651815 14768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33745 (error 108)
I20260924 11:43:51.653544 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14971
I20260924 11:43:51.655023 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.654511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42656)
0924 11:43:51.654735 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.654843 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260924 11:43:51.655676 14768 catalog_manager.cc:4584] TS 62055ee072fa40b5ba835375ee3233e2 (127.0.97.1:33745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a3ede37f118405fa0acacc0b437b7a4: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
I20260924 11:43:51.672206 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.655502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.0.97.2:45799)
0924 11:43:51.655655 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.655663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.655682 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:43:51.655747 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.655752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.655843 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.655938 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:51.658388 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.659159 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.661288 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.661939 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.662070 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.665463 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:51.665479 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:51.665497 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:51.665549 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:51.669191 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.669948 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.669960 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.669964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.670125 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.670502 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.670508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.670514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.670981 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:51.671251 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.671512 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.671759 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 11:43:51.672916 15094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:51.709344 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.708868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42670)
0924 11:43:51.709070 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.709172 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 11:43:51.710407 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.710064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42674)
0924 11:43:51.710209 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.710296 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:43:51.763245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14971
I20260924 11:43:51.770993 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.770384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42682)
0924 11:43:51.770676 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.770794 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 11:43:51.772229 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.771769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42690)
0924 11:43:51.771963 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.772070 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:43:51.781046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 15102
I20260924 11:43:51.800400 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.781772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.0.97.3:33125)
0924 11:43:51.781976 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:43:51.781983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:43:51.782229 (+ 246us) server_negotiation.cc:407] Connection header received
0924 11:43:51.782439 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:43:51.782443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:43:51.782549 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:43:51.782648 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:43:51.787901 (+ 5253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.789054 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.790836 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:43:51.791279 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:43:51.791369 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:43:51.794449 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:43:51.794463 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:43:51.794481 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:43:51.794522 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:43:51.797564 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.798289 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.798295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.798297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.798431 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:43:51.798818 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:43:51.798821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:43:51.798822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:43:51.799147 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:43:51.799369 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:43:51.799716 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:43:51.799861 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:43:51.801131 15225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:51.852510 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.852006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42700)
0924 11:43:51.852218 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.852331 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:43:51.853524 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:51.853133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42712)
0924 11:43:51.853314 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:51.853401 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 11:43:51.915635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 15102
I20260924 11:43:51.935043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 14749
I20260924 11:43:51.936086 14812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:43:52.038584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 14749
I20260924 11:43:52.038681 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:52.037850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33745 (local address 127.0.0.1:42720)
0924 11:43:52.038430 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:52.038524 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456}
I20260924 11:43:52.058962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35925
--webserver_interface=127.0.97.62
--webserver_port=44993
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:52.355273 15276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:52.355489 15276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:52.355567 15276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:52.362033 15276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:43:52.362118 15276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:52.362210 15276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:52.362306 15276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:43:52.362387 15276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:43:52.370935 15276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:52.371975 15276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:52.373111 15276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:52.384267 15281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:52.384743 15282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:52.385677 15284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:52.386103 15276 server_base.cc:1034] running on GCE node
I20260924 11:43:52.386812 15276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:52.387970 15276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:52.389183 15276 hybrid_clock.cc:648] HybridClock initialized: now 1790250232389147 us; error 39 us; skew 500 ppm
I20260924 11:43:52.389492 15276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:52 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:52.406320 15276 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:43:52.407546 15276 webserver.cc:460] Webserver started at http://127.0.97.62:44993/ using document root <none> and password file <none>
I20260924 11:43:52.408133 15276 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:52.408309 15276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:52.412942 15276 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:43:52.415431 15291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:52.416582 15276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:43:52.416744 15276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "2f30d11e5e45413db6dda133ac2004b8"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "957e4840b73ff46f3b9f260d90265cf5"
server_key_iv: "390b237bac74d28dc8f0fa762eb18d65"
server_key_version: "encryptionkey@0"
I20260924 11:43:52.417570 15276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:52.440428 15276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:52.443847 15276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:52.444120 15276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:52.461220 15276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35925
I20260924 11:43:52.461298 15342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35925 every 8 connection(s)
I20260924 11:43:52.462543 15276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:43:52.468606 15343 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:43:52.468875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15276
I20260924 11:43:52.469643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33745
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35925
--webserver_port=44253
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:43:52.471920 15343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap starting.
I20260924 11:43:52.490561 15343 log.cc:824] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:43:52.505038 15343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:43:52.505508 15343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8: Bootstrap complete.
I20260924 11:43:52.513921 15343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:52.514734 15343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f30d11e5e45413db6dda133ac2004b8, State: Initialized, Role: FOLLOWER
I20260924 11:43:52.515359 15343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:52.515575 15343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:43:52.515698 15343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:43:52.515870 15343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:43:52.519178 15343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:52.519618 15343 leader_election.cc:302] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f30d11e5e45413db6dda133ac2004b8; no voters:
I20260924 11:43:52.520290 15343 leader_election.cc:288] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 11:43:52.520429 15347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:43:52.522555 15343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:43:52.524415 15347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [term 3 LEADER]: Becoming Leader. State: Replica: 2f30d11e5e45413db6dda133ac2004b8, State: Running, Role: LEADER
I20260924 11:43:52.525029 15347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } }
I20260924 11:43:52.527446 15349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f30d11e5e45413db6dda133ac2004b8. Latest consensus state: current_term: 3 leader_uuid: "2f30d11e5e45413db6dda133ac2004b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } } }
I20260924 11:43:52.527817 15349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:52.528101 15348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2f30d11e5e45413db6dda133ac2004b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f30d11e5e45413db6dda133ac2004b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35925 } } }
I20260924 11:43:52.528405 15348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f30d11e5e45413db6dda133ac2004b8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:43:52.541054 15356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:43:52.548938 15356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:43:52.550243 15356 catalog_manager.cc:1259] Loaded cluster ID: 40f269c04f714abd949f3dac8c9b399f
I20260924 11:43:52.550349 15356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 11:43:52.552031 15356 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff9252aa9ad google::LogMessage::Fail() at ??:0
@ 0x7ff9252ad88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff9252aa4de google::LogMessage::Flush() at ??:0
@ 0x7ff9252aba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff93acdd512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff93acda976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff93ad62610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff93ad62439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff927ef6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff927ef991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff927ef9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff927ed4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff92b1ae6db start_thread at ??:0
@ 0x7ff92414461f clone at ??:0
W20260924 11:43:52.831437 15345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:52.831669 15345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:52.831755 15345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:52.838479 15345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:52.838560 15345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:52.838672 15345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:43:52.847604 15345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44253
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:52.848717 15345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:52.849839 15345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:52.862243 15384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:52.862314 15387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:52.862277 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:52.862979 15345 server_base.cc:1034] running on GCE node
I20260924 11:43:52.863510 15345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:52.864408 15345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:52.865592 15345 hybrid_clock.cc:648] HybridClock initialized: now 1790250232865542 us; error 52 us; skew 500 ppm
I20260924 11:43:52.865888 15345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:52 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:52.868809 15345 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:43:52.870127 15345 webserver.cc:460] Webserver started at http://127.0.97.1:44253/ using document root <none> and password file <none>
I20260924 11:43:52.870761 15345 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:52.870858 15345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:52.875459 15345 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 11:43:52.878173 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:52.879331 15345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:43:52.879451 15345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "62055ee072fa40b5ba835375ee3233e2"
format_stamp: "Formatted at 2026-09-24 11:43:47 on dist-test-slave-qhrh"
server_key: "b20f0016f3baa7842166b78d99660885"
server_key_iv: "936cac53dff84e69af2a556fa4060e66"
server_key_version: "encryptionkey@0"
I20260924 11:43:52.880371 15345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:52.898401 15345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:52.902527 15345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:52.902731 15345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:52.903563 15345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:52.905169 15401 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 11:43:52.910298 15401 ts_tablet_manager.cc:1852] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 11:43:52.911260 15401 ts_tablet_manager.cc:1905] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:43:52.917958 15401 ts_tablet_manager.cc:1918] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:43:52.918140 15401 log.cc:1196] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/8a3ede37f118405fa0acacc0b437b7a4
I20260924 11:43:52.918602 15401 ts_tablet_manager.cc:1939] T 8a3ede37f118405fa0acacc0b437b7a4 P 62055ee072fa40b5ba835375ee3233e2: Deleting consensus metadata
I20260924 11:43:52.918972 15345 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 11:43:52.919062 15345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260924 11:43:52.919220 15345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:52.919306 15345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:52.950599 15345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33745
I20260924 11:43:52.950675 15507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33745 every 8 connection(s)
I20260924 11:43:52.951815 15345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:43:52.956130 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:52.954528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:52619)
0924 11:43:52.955176 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:52.955363 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":257,"threads_started":1}
W20260924 11:43:52.956707 15508 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35925 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35925: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
I20260924 11:43:52.957351 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:52.957109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:36853)
0924 11:43:52.957214 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:52.957268 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 11:43:52.958084 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:52.957832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:54249)
0924 11:43:52.957931 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:52.957990 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 11:43:52.958359 15508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:43:52.961906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15345
I20260924 11:43:52.962463 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45799
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35925
--webserver_port=45043
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:53.263242 15511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:53.263470 15511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:53.263623 15511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:53.270319 15511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:53.270414 15511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:53.270609 15511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:43:53.279732 15511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45043
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:53.280910 15511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:53.282148 15511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:53.293879 15516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:53.293962 15517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:53.293915 15519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:53.294661 15511 server_base.cc:1034] running on GCE node
I20260924 11:43:53.295200 15511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:53.296067 15511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:53.297294 15511 hybrid_clock.cc:648] HybridClock initialized: now 1790250233297245 us; error 49 us; skew 500 ppm
I20260924 11:43:53.297586 15511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:53 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:53.300499 15511 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:43:53.301880 15511 webserver.cc:460] Webserver started at http://127.0.97.2:45043/ using document root <none> and password file <none>
I20260924 11:43:53.302511 15511 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:53.302646 15511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:53.307466 15511 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:43:53.311923 15526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:53.313031 15511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:43:53.313180 15511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "c698cf01febc437bba468e3f996726f1"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "fd52279d8d26d22b9c66e4a3d8d3aebf"
server_key_iv: "1ecbac803d7d99a24924ef4cafc952c6"
server_key_version: "encryptionkey@0"
I20260924 11:43:53.314072 15511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:53.327490 15511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:53.331166 15511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:53.331477 15511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:53.332333 15511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:53.333417 15511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:53.333531 15511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:53.333648 15511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:53.333726 15511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:53.363487 15511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45799
I20260924 11:43:53.363556 15638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45799 every 8 connection(s)
I20260924 11:43:53.364791 15511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:43:53.368810 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.367264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:55657)
0924 11:43:53.367860 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.368037 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":238,"threads_started":1}
W20260924 11:43:53.369376 15639 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35925 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35925: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
I20260924 11:43:53.370087 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.369773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:34295)
0924 11:43:53.369946 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.370002 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 11:43:53.370776 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.370496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:40161)
0924 11:43:53.370616 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.370696 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 11:43:53.370965 15639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:43:53.372224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15511
I20260924 11:43:53.372762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33125
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35925
--webserver_port=38175
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:43:53.664997 15642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:43:53.665223 15642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:43:53.665423 15642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:43:53.671700 15642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:43:53.671816 15642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:43:53.672011 15642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:43:53.680212 15642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38175
--tserver_master_addrs=127.0.97.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:43:53.681169 15642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:43:53.682262 15642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:43:53.694113 15650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:53.694196 15647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:43:53.694175 15648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:43:53.695042 15642 server_base.cc:1034] running on GCE node
I20260924 11:43:53.695573 15642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:43:53.696395 15642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:43:53.697576 15642 hybrid_clock.cc:648] HybridClock initialized: now 1790250233697519 us; error 56 us; skew 500 ppm
I20260924 11:43:53.697851 15642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:43:53 dist-test-slave-qhrh krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:43:53.700573 15642 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:43:53.701856 15642 webserver.cc:460] Webserver started at http://127.0.97.3:38175/ using document root <none> and password file <none>
I20260924 11:43:53.702462 15642 fs_manager.cc:359] Metadata directory not provided
I20260924 11:43:53.702584 15642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:43:53.707252 15642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 11:43:53.709722 15657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:53.710882 15642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:43:53.711037 15642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "7c7b3b23c7274c8b8f1b98bf402afbe3"
format_stamp: "Formatted at 2026-09-24 11:43:48 on dist-test-slave-qhrh"
server_key: "a9aa9bc6555a461cdcaa2d623139758a"
server_key_iv: "6f0253b4598f7e2e7c40f84c30da3140"
server_key_version: "encryptionkey@0"
I20260924 11:43:53.711884 15642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:43:53.740780 15642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:43:53.744443 15642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:43:53.744689 15642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:43:53.745523 15642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:43:53.746569 15642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:43:53.746677 15642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:53.746789 15642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:43:53.746874 15642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:43:53.776374 15642 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33125
I20260924 11:43:53.776441 15769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33125 every 8 connection(s)
I20260924 11:43:53.777576 15642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:43:53.781677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15642
I20260924 11:43:53.781800 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.780037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:52105)
0924 11:43:53.780782 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.780948 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":310,"threads_started":1}
W20260924 11:43:53.782512 15770 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35925 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35925: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
I20260924 11:43:53.783224 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.782968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:36449)
0924 11:43:53.783066 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.783118 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 11:43:53.783985 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.783720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:43681)
0924 11:43:53.783828 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.783889 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 11:43:53.784325 15770 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:43:53.959625 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:53.958869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:39747)
0924 11:43:53.959368 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:53.959438 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":301,"threads_started":1}
I20260924 11:43:54.372285 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:54.371577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:60115)
0924 11:43:54.371968 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:54.372037 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260924 11:43:54.785485 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:54.784763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:46185)
0924 11:43:54.785159 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:54.785252 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":196,"threads_started":1}
I20260924 11:43:54.961107 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:54.960499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:54519)
0924 11:43:54.960867 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:54.960933 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":200,"threads_started":1}
I20260924 11:43:55.373661 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:55.373030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:51175)
0924 11:43:55.373437 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:55.373516 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":204,"threads_started":1}
I20260924 11:43:55.786804 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:55.786217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:55559)
0924 11:43:55.786611 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:55.786675 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
I20260924 11:43:55.962579 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:55.961858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:39325)
0924 11:43:55.962327 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:55.962426 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":198,"threads_started":1}
I20260924 11:43:56.374989 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:56.374414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:46991)
0924 11:43:56.374778 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:56.374846 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":191,"threads_started":1}
I20260924 11:43:56.788141 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:56.787578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:39359)
0924 11:43:56.787937 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:56.788012 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":196,"threads_started":1}
I20260924 11:43:56.964068 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:56.963497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:48679)
0924 11:43:56.963850 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:56.963921 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":180,"threads_started":1}
I20260924 11:43:57.376514 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:57.375745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:50687)
0924 11:43:57.376257 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:57.376349 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":263,"threads_started":1}
I20260924 11:43:57.789650 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:57.788939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:48191)
0924 11:43:57.789382 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:57.789446 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":246,"threads_started":1}
I20260924 11:43:57.965482 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:57.964866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:38359)
0924 11:43:57.965249 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:57.965328 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":195,"threads_started":1}
I20260924 11:43:58.377883 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:58.377277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:34705)
0924 11:43:58.377679 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:58.377744 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":205,"threads_started":1}
I20260924 11:43:58.791077 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:58.790454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:47553)
0924 11:43:58.790885 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:58.790949 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":259,"threads_started":1}
I20260924 11:43:58.966863 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:58.966292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:36669)
0924 11:43:58.966667 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:58.966733 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":207,"threads_started":1}
I20260924 11:43:59.379261 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:59.378674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:59745)
0924 11:43:59.379056 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:59.379127 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":195,"threads_started":1}
I20260924 11:43:59.792359 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:59.791722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:53925)
0924 11:43:59.792094 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:59.792212 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":203,"threads_started":1}
I20260924 11:43:59.968410 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:43:59.967713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:54011)
0924 11:43:59.968117 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 11:43:59.968233 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":193,"threads_started":1}
I20260924 11:44:00.380705 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:00.380005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:58455)
0924 11:44:00.380466 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:00.380530 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
I20260924 11:44:00.793823 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:00.793129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:60567)
0924 11:44:00.793569 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:00.793643 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":195,"threads_started":1}
I20260924 11:44:00.969823 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:00.969190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:55867)
0924 11:44:00.969566 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:00.969635 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":204,"threads_started":1}
I20260924 11:44:01.382287 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:01.381662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:50683)
0924 11:44:01.382067 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:01.382137 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":205,"threads_started":1}
I20260924 11:44:01.795174 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:01.794578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:60877)
0924 11:44:01.794968 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:01.795037 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":194,"threads_started":1}
I20260924 11:44:01.971534 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:01.970843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:44909)
0924 11:44:01.971287 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:01.971378 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260924 11:44:02.383772 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:02.383102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:47019)
0924 11:44:02.383533 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:02.383598 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
I20260924 11:44:02.796749 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:02.795964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:43627)
0924 11:44:02.796524 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:02.796595 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":341,"threads_started":1}
I20260924 11:44:02.972930 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:02.972374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:33775)
0924 11:44:02.972727 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:02.972794 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":189,"threads_started":1}
I20260924 11:44:03.385305 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:03.384610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:34831)
0924 11:44:03.385080 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:03.385156 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":286,"threads_started":1}
I20260924 11:44:03.798163 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:03.797544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:36243)
0924 11:44:03.797942 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:03.798010 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":237,"threads_started":1}
I20260924 11:44:03.974275 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:03.973650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:49513)
0924 11:44:03.974043 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:03.974113 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260924 11:44:04.386687 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:04.386074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:44231)
0924 11:44:04.386446 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:04.386523 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":183,"threads_started":1}
I20260924 11:44:04.799670 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:04.799053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:54067)
0924 11:44:04.799430 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:04.799508 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":185,"threads_started":1}
I20260924 11:44:04.975872 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:04.975270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:56981)
0924 11:44:04.975648 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:04.975719 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":204,"threads_started":1}
I20260924 11:44:05.388177 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:05.387586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:44319)
0924 11:44:05.387948 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:05.388016 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":187,"threads_started":1}
I20260924 11:44:05.801221 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:05.800607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:53819)
0924 11:44:05.801005 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:05.801078 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":222,"threads_started":1}
I20260924 11:44:05.977660 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:05.976880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:43833)
0924 11:44:05.977397 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:05.977477 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
I20260924 11:44:06.389930 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:06.389151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:59095)
0924 11:44:06.389695 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:06.389764 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":293,"threads_started":1}
I20260924 11:44:06.802696 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:06.802055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:49417)
0924 11:44:06.802454 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:06.802531 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":199,"threads_started":1}
I20260924 11:44:06.979180 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:06.978566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:52377)
0924 11:44:06.978948 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:06.979017 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":186,"threads_started":1}
I20260924 11:44:07.391327 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:07.390705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:56777)
0924 11:44:07.391103 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:07.391171 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":201,"threads_started":1}
I20260924 11:44:07.804574 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:07.803873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:39053)
0924 11:44:07.804347 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:07.804415 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":278,"threads_started":1}
I20260924 11:44:07.980834 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:07.980109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.1:42881)
0924 11:44:07.980569 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:07.980652 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":212,"threads_started":1}
I20260924 11:44:08.393046 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:08.392399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.2:35917)
0924 11:44:08.392814 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:08.392890 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260924 11:44:08.791115 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 15345
I20260924 11:44:08.806258 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:08.805539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35925 (local address 127.0.97.3:41817)
0924 11:44:08.806001 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:08.806092 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":248,"threads_started":1}
I20260924 11:44:08.819235 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:08.792144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.0.97.1:33745)
0924 11:44:08.793178 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:08.793191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:08.793223 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:44:08.794872 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:08.794882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:08.795299 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:08.795653 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:08.797039 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:08.798200 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:08.799384 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:08.800031 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:08.800256 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:08.813160 (+ 12904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:08.813167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:08.813181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:08.813215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:08.816047 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:08.816770 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:08.816775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:08.816782 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:08.816937 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:08.817322 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:08.817326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:08.817328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:08.817874 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:08.818170 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:08.818580 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 11:44:08.818966 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":766,"threads_started":1}
I20260924 11:44:08.820521 15503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:08.900902 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 15345
I20260924 11:44:08.917358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 15511
I20260924 11:44:08.933151 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:08.918327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.0.97.2:45799)
0924 11:44:08.918810 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:08.918841 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:08.918877 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:44:08.920453 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:08.920462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:08.920868 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:08.921143 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:08.922330 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:08.923482 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:08.924649 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:08.925289 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:08.925423 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:08.927762 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:08.927769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:08.927782 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:08.927819 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:08.930471 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:08.931013 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:08.931018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:08.931024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:08.931173 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:08.931473 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:08.931476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:08.931478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:08.931909 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:08.932118 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:08.932608 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 11:44:08.932893 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260924 11:44:08.934396 15634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:09.016405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 15511
I20260924 11:44:09.034020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 15642
I20260924 11:44:09.052618 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:09.034872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.0.97.3:33125)
0924 11:44:09.035477 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:09.035492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:09.035535 (+ 43us) server_negotiation.cc:407] Connection header received
0924 11:44:09.037260 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:09.037270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:09.037704 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:09.038060 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:09.039657 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:09.041013 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:09.042376 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:09.043014 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:09.043205 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:09.046045 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:09.046053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:09.046067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:09.046124 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:09.049197 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:09.049923 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:09.049938 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:09.049945 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:09.050103 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:09.050542 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:09.050549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:09.050552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:09.051089 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:09.051397 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:09.051861 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 11:44:09.052311 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":283,"threads_started":1}
I20260924 11:44:09.053972 15765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:09.134276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 15642
2026-09-24T11:44:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23630 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:44:09 dist-test-slave-qhrh krb5kdc[15827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:44:09 dist-test-slave-qhrh krb5kdc[15827](info): set up 2 sockets
Sep 24 11:44:09 dist-test-slave-qhrh krb5kdc[15827](info): commencing operation
krb5kdc: starting...
W20260924 11:44:10.176678 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.010s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:44:10 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:44:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:44:10.388111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37105
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:44:10.696652 15840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:10.696923 15840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:10.697007 15840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:10.703677 15840 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:44:10.703750 15840 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:44:10.703783 15840 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:44:10.703833 15840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:44:10.703879 15840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:10.703944 15840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:10.703985 15840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:44:10.704026 15840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:44:10.712839 15840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:10.713958 15840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:10.715180 15840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:10.726434 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:10.726526 15846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:10.726477 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:10.727054 15840 server_base.cc:1034] running on GCE node
I20260924 11:44:10.727861 15840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:10.729197 15840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:10.730437 15840 hybrid_clock.cc:648] HybridClock initialized: now 1790250250730398 us; error 56 us; skew 500 ppm
I20260924 11:44:10.730757 15840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:10 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:10.733835 15840 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:44:10.735087 15840 webserver.cc:460] Webserver started at http://127.0.97.62:38751/ using document root <none> and password file <none>
I20260924 11:44:10.735687 15840 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:10.735769 15840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:10.736112 15840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:44:10.738875 15840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "93a29c0f4d654c70ba8eec99c0d0739a"
format_stamp: "Formatted at 2026-09-24 11:44:10 on dist-test-slave-qhrh"
server_key: "bfc5b5cd54cc88350049a3de639e7095"
server_key_iv: "cafb79c6cf4b5e1745025b15adc11478"
server_key_version: "encryptionkey@0"
I20260924 11:44:10.739617 15840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "93a29c0f4d654c70ba8eec99c0d0739a"
format_stamp: "Formatted at 2026-09-24 11:44:10 on dist-test-slave-qhrh"
server_key: "bfc5b5cd54cc88350049a3de639e7095"
server_key_iv: "cafb79c6cf4b5e1745025b15adc11478"
server_key_version: "encryptionkey@0"
I20260924 11:44:10.744372 15840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260924 11:44:10.747489 15855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:10.748653 15840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:44:10.748842 15840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "93a29c0f4d654c70ba8eec99c0d0739a"
format_stamp: "Formatted at 2026-09-24 11:44:10 on dist-test-slave-qhrh"
server_key: "bfc5b5cd54cc88350049a3de639e7095"
server_key_iv: "cafb79c6cf4b5e1745025b15adc11478"
server_key_version: "encryptionkey@0"
I20260924 11:44:10.749053 15840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:10.777896 15840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:10.781464 15840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:10.781723 15840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:10.799187 15840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37105
I20260924 11:44:10.799216 15906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37105 every 8 connection(s)
I20260924 11:44:10.800479 15840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:44:10.803848 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:10.807091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15840
I20260924 11:44:10.807291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:44:10.808018 388 external_mini_cluster.cc:1428] Setting key 95ef9fe77ee6a21f2a6389f449b45abf
I20260924 11:44:10.816203 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap starting.
I20260924 11:44:10.819707 15907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:10.820992 15907 log.cc:824] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:44:10 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:10.823663 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: No bootstrap required, opened a new log
I20260924 11:44:10.830435 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:10.809904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.0.97.62:37105)
0924 11:44:10.810617 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:10.810627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:10.810665 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:44:10.812370 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:10.812390 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:10.812740 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:10.813051 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:10.814788 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:10.816015 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:10.817824 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:10.818431 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:10.818550 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:10.823398 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:10.823404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:10.823416 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:10.823449 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:10.826621 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:10.827284 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:10.827296 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:10.827301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:10.827460 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:10.827877 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:10.827882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:10.827885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:10.828424 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:10.828741 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:10.829159 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 11:44:10.829547 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
I20260924 11:44:10.831300 15907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:10.831648 15907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:10.831750 15907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a29c0f4d654c70ba8eec99c0d0739a, State: Initialized, Role: FOLLOWER
I20260924 11:44:10.832361 15907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:10.832592 15907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:44:10.832738 15907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:44:10.832909 15907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:10.834862 15907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:10.835317 15907 leader_election.cc:302] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a29c0f4d654c70ba8eec99c0d0739a; no voters:
I20260924 11:44:10.835827 15907 leader_election.cc:288] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:44:10.835966 15912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:44:10.836881 15912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 LEADER]: Becoming Leader. State: Replica: 93a29c0f4d654c70ba8eec99c0d0739a, State: Running, Role: LEADER
I20260924 11:44:10.837399 15912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:10.837841 15907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:44:10.840131 15913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } } }
I20260924 11:44:10.840378 15914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a29c0f4d654c70ba8eec99c0d0739a. Latest consensus state: current_term: 1 leader_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } } }
I20260924 11:44:10.840655 15913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: This master's current role is: LEADER
I20260924 11:44:10.840688 15914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: This master's current role is: LEADER
I20260924 11:44:10.844200 15920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:44:10.849000 15920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:44:10.857018 15920 catalog_manager.cc:1347] Generated new cluster ID: a11a2939b90e4c6f96af3aed8f903b04
I20260924 11:44:10.857131 15920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:44:10.871783 15920 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:44:10.872941 15920 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:44:10.880700 15920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Generated new TSK 0
I20260924 11:44:10.881559 15920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:44:10.944583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37105
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:44:11.259123 15933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:11.259382 15933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:11.259527 15933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:11.266454 15933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:11.266546 15933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:11.266738 15933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:44:11.276188 15933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:11.277431 15933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:11.278628 15933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:11.290987 15938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:11.291019 15939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:11.291051 15941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:11.291786 15933 server_base.cc:1034] running on GCE node
I20260924 11:44:11.292371 15933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:11.293174 15933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:11.294360 15933 hybrid_clock.cc:648] HybridClock initialized: now 1790250251294306 us; error 53 us; skew 500 ppm
I20260924 11:44:11.294641 15933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:11 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:11.297622 15933 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:44:11.298883 15933 webserver.cc:460] Webserver started at http://127.0.97.1:41701/ using document root <none> and password file <none>
I20260924 11:44:11.299522 15933 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:11.299662 15933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:11.299969 15933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:44:11.302798 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "14056ec8151847e88ec570fcd96ec5c0"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "5b83435e39b917b546d53a564a59b465"
server_key_iv: "f0b2a13993e81bf229eb36caa010699c"
server_key_version: "encryptionkey@0"
I20260924 11:44:11.303547 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "14056ec8151847e88ec570fcd96ec5c0"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "5b83435e39b917b546d53a564a59b465"
server_key_iv: "f0b2a13993e81bf229eb36caa010699c"
server_key_version: "encryptionkey@0"
I20260924 11:44:11.308346 15933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260924 11:44:11.311239 15948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:11.312388 15933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:44:11.312534 15933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "14056ec8151847e88ec570fcd96ec5c0"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "5b83435e39b917b546d53a564a59b465"
server_key_iv: "f0b2a13993e81bf229eb36caa010699c"
server_key_version: "encryptionkey@0"
I20260924 11:44:11.312704 15933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:11.333959 15933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:11.337814 15933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:11.338076 15933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:11.338922 15933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:11.340071 15933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:11.340193 15933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:11.340291 15933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:11.340333 15933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:11.371151 15933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33257
I20260924 11:44:11.371229 16060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33257 every 8 connection(s)
I20260924 11:44:11.372504 15933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:44:11.376600 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 15933
I20260924 11:44:11.376811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:44:11.377461 388 external_mini_cluster.cc:1428] Setting key 71a9697413933d9f6cff107c60739e4f
Sep 24 11:44:11 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:11.392076 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:11.374872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59043 (local address 127.0.97.62:37105)
0924 11:44:11.375326 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:11.375331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:11.376082 (+ 751us) server_negotiation.cc:407] Connection header received
0924 11:44:11.378119 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:11.378124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:11.378215 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:11.378331 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:11.379687 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.380518 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.382094 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.382478 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.382599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:11.386055 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:11.386060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:11.386063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:11.386094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:11.389021 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:11.389654 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:11.389661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:11.389666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:11.389800 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:11.390221 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:11.390228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:11.390234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:11.390591 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:11.390740 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:11.391354 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 11:44:11.391600 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":249,"threads_started":1}
I20260924 11:44:11.392611 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:11.375271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:59043)
0924 11:44:11.375937 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:11.375972 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:11.377833 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:11.378558 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:11.378565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:11.379021 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:44:11.379525 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:11.379544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.380667 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.380671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:11.381960 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:11.381971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.382691 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.382697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:11.382903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:11.383573 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:44:11.383593 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:44:11.385843 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:44:11.389168 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:11.389174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:11.389183 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:11.389540 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:11.389933 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:11.389936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:11.389938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:11.390115 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:11.390704 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:44:11.390709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:44:11.390975 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 11:44:11.391216 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 11:44:11.391546 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":258,"threads_started":1}
I20260924 11:44:11.394114 16061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37105
I20260924 11:44:11.394414 16061 heartbeater.cc:464] Registering TS with master...
I20260924 11:44:11.395215 16061 heartbeater.cc:511] Master 127.0.97.62:37105 requested a full tablet report, sending...
I20260924 11:44:11.397269 15872 ts_manager.cc:194] Registered new tserver with Master: 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257)
I20260924 11:44:11.398696 15872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59043
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:44:11.427260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37105
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:44:11.729321 16067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:11.729554 16067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:11.729694 16067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:11.736559 16067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:11.736671 16067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:11.736842 16067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:44:11.745939 16067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:11.747072 16067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:11.748369 16067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:11.765496 16072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:11.765540 16073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:11.766251 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:11.767088 16067 server_base.cc:1034] running on GCE node
I20260924 11:44:11.767830 16067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:11.768970 16067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:11.770195 16067 hybrid_clock.cc:648] HybridClock initialized: now 1790250251770136 us; error 68 us; skew 500 ppm
I20260924 11:44:11.770527 16067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:11 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:11.774639 16067 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:44:11.776497 16067 webserver.cc:460] Webserver started at http://127.0.97.2:40319/ using document root <none> and password file <none>
I20260924 11:44:11.777413 16067 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:11.777547 16067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:11.777906 16067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:44:11.782274 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "eda79c7217e54f9e8bde533abf1a5e3a"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "dacfeae78a3c18c3d10865d7dccf8ea6"
server_key_iv: "38999ea1922b27187835d16e495e5c2b"
server_key_version: "encryptionkey@0"
I20260924 11:44:11.783388 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eda79c7217e54f9e8bde533abf1a5e3a"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "dacfeae78a3c18c3d10865d7dccf8ea6"
server_key_iv: "38999ea1922b27187835d16e495e5c2b"
server_key_version: "encryptionkey@0"
I20260924 11:44:11.789094 16067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:44:11.792462 16082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:11.793794 16067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:44:11.793954 16067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "eda79c7217e54f9e8bde533abf1a5e3a"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "dacfeae78a3c18c3d10865d7dccf8ea6"
server_key_iv: "38999ea1922b27187835d16e495e5c2b"
server_key_version: "encryptionkey@0"
I20260924 11:44:11.794147 16067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:11.841430 16067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:11.845790 16067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:11.846025 16067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:11.846869 16067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:11.848012 16067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:11.848093 16067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:11.848240 16067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:11.848316 16067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 11:44:11.878746 16067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33029
I20260924 11:44:11.878785 16194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33029 every 8 connection(s)
I20260924 11:44:11.879855 16067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:44:11.886368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16067
I20260924 11:44:11.886559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:44:11.887028 388 external_mini_cluster.cc:1428] Setting key f0e5c0cda01632e9fb224ffdf6e5a48c
Sep 24 11:44:11 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:11.900203 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:11.882347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60291 (local address 127.0.97.62:37105)
0924 11:44:11.882541 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:11.882547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:11.883866 (+ 1319us) server_negotiation.cc:407] Connection header received
0924 11:44:11.885640 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:11.885644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:11.885712 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:11.885799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:11.887449 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.888189 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.889599 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.890003 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.890126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:11.893860 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:11.893865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:11.893869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:11.893897 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:11.896935 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:11.897705 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:11.897713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:11.897719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:11.897865 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:11.898375 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:11.898383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:11.898388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:11.898713 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:11.898859 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:11.899458 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 11:44:11.899732 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:44:11.900851 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:11.882875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:60291)
0924 11:44:11.883724 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:11.883767 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:11.885405 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:11.885985 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:11.885991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:11.886667 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:44:11.887294 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:11.887316 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.888344 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.888348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:11.889466 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:11.889477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:11.890200 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:11.890206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:11.890418 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:11.891138 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:44:11.891159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:44:11.893650 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:44:11.897074 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:11.897081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:11.897095 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:11.897595 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:11.898000 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:11.898003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:11.898005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:11.898249 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:11.898829 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:44:11.898834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:44:11.899124 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 11:44:11.899342 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 11:44:11.899737 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":346,"threads_started":1}
I20260924 11:44:11.902171 16195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37105
I20260924 11:44:11.902496 16195 heartbeater.cc:464] Registering TS with master...
I20260924 11:44:11.903398 16195 heartbeater.cc:511] Master 127.0.97.62:37105 requested a full tablet report, sending...
I20260924 11:44:11.904783 15872 ts_manager.cc:194] Registered new tserver with Master: eda79c7217e54f9e8bde533abf1a5e3a (127.0.97.2:33029)
I20260924 11:44:11.905622 15872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60291
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:44:11.936609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37105
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:44:12.241150 16200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:12.241436 16200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:12.241595 16200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:12.248531 16200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:12.248672 16200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:12.248822 16200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:44:12.258034 16200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:12.259227 16200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:12.260520 16200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:12.275375 16205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:12.275432 16206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:12.275635 16208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:12.276345 16200 server_base.cc:1034] running on GCE node
I20260924 11:44:12.276882 16200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:12.277758 16200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:12.279006 16200 hybrid_clock.cc:648] HybridClock initialized: now 1790250252278954 us; error 58 us; skew 500 ppm
I20260924 11:44:12.279312 16200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:12 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:12.282537 16200 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:44:12.283892 16200 webserver.cc:460] Webserver started at http://127.0.97.3:35591/ using document root <none> and password file <none>
I20260924 11:44:12.284555 16200 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:12.284682 16200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:12.284974 16200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:44:12.287832 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "4550cb304917423588a7db6e39d8d9f0"
format_stamp: "Formatted at 2026-09-24 11:44:12 on dist-test-slave-qhrh"
server_key: "d8bb14beb7b3e283ea6f5ec7c14a1047"
server_key_iv: "71877c2a70e037296b1935f88f44311a"
server_key_version: "encryptionkey@0"
I20260924 11:44:12.288556 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4550cb304917423588a7db6e39d8d9f0"
format_stamp: "Formatted at 2026-09-24 11:44:12 on dist-test-slave-qhrh"
server_key: "d8bb14beb7b3e283ea6f5ec7c14a1047"
server_key_iv: "71877c2a70e037296b1935f88f44311a"
server_key_version: "encryptionkey@0"
I20260924 11:44:12.293370 16200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:44:12.296664 16215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:12.298041 16200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:44:12.298254 16200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "4550cb304917423588a7db6e39d8d9f0"
format_stamp: "Formatted at 2026-09-24 11:44:12 on dist-test-slave-qhrh"
server_key: "d8bb14beb7b3e283ea6f5ec7c14a1047"
server_key_iv: "71877c2a70e037296b1935f88f44311a"
server_key_version: "encryptionkey@0"
I20260924 11:44:12.298527 16200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:12.315881 16200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:12.320206 16200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:12.320514 16200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:12.321491 16200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:12.322784 16200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:12.322885 16200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:12.323029 16200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:12.323101 16200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:12.356426 16200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35675
I20260924 11:44:12.356468 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35675 every 8 connection(s)
I20260924 11:44:12.357715 16200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:44:12.367892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16200
I20260924 11:44:12.368078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:44:12.368691 388 external_mini_cluster.cc:1428] Setting key f2913e949d99c8a9c04574edeb603a6d
Sep 24 11:44:12 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:12.377101 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.360323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55471 (local address 127.0.97.62:37105)
0924 11:44:12.360535 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.360540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.361685 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 11:44:12.363392 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.363396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.363461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.363548 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:12.365028 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.365762 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.367161 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.367524 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.367614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.371442 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:12.371447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:12.371450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:12.371506 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:12.374123 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.374799 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.374806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.374814 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.374940 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.375378 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.375386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.375388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.375643 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:12.375801 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.376515 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.376751 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:44:12.377918 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.360830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:55471)
0924 11:44:12.361523 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:12.361560 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:12.363153 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:12.363743 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:12.363751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:12.364318 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:44:12.364884 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.364901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.365893 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.365896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.367045 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.367055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.367751 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.367759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.368058 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.368744 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:44:12.368780 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:44:12.371204 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:44:12.374263 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:12.374271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:12.374286 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:12.374673 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:12.375048 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:12.375053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:12.375057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:12.375255 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:12.375768 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:44:12.375776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:44:12.376078 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 11:44:12.376410 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 11:44:12.376908 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
I20260924 11:44:12.379125 16328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37105
I20260924 11:44:12.379369 16328 heartbeater.cc:464] Registering TS with master...
I20260924 11:44:12.379973 16328 heartbeater.cc:511] Master 127.0.97.62:37105 requested a full tablet report, sending...
I20260924 11:44:12.381312 15872 ts_manager.cc:194] Registered new tserver with Master: 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675)
I20260924 11:44:12.382115 15872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55471
I20260924 11:44:12.383867 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:44:12.399838 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.387996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.0.97.62:37105)
0924 11:44:12.388197 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.388202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.388653 (+ 451us) server_negotiation.cc:407] Connection header received
0924 11:44:12.388844 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.388849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.388910 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.389007 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:12.390288 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.390922 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.392205 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.392607 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.392737 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.394986 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:12.394991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:12.394994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:12.395021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:12.397419 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.398034 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.398040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.398043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.398166 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.398519 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.398524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.398527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.398794 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:12.399044 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.399337 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.399552 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:44:12.402036 16061 heartbeater.cc:503] Master 127.0.97.62:37105 was elected leader, sending a full tablet report...
I20260924 11:44:12.403671 15871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:44:12.410084 15871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:44:12.443557 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.421257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35675 (local address 127.0.0.1:39172)
0924 11:44:12.435025 (+ 13768us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:12.435047 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:12.435222 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:12.436254 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:12.436261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:12.436302 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:12.436667 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.436703 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.438825 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.438829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.440923 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.440934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.442546 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.442552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.442746 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.442821 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 11:44:12.442940 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 11:44:12.443110 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13622,"mutex_wait_us":8,"thread_start_us":14797,"threads_started":1}
I20260924 11:44:12.443984 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.421828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.0.97.3:35675)
0924 11:44:12.434719 (+ 12891us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.434729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.435194 (+ 465us) server_negotiation.cc:407] Connection header received
0924 11:44:12.435405 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.435416 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.435689 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.435926 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:12.436925 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.438675 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.441086 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.442360 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.442491 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.442748 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.443029 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.443195 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12716,"thread_start_us":12573,"threads_started":1}
I20260924 11:44:12.445554 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.423418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:51506)
0924 11:44:12.436451 (+ 13033us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:12.436473 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:12.436630 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:12.437457 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:12.437462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:12.437485 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:12.437854 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.437864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.439442 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.439447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.441462 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.441476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.444614 (+ 3138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.444622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.444815 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.444909 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 11:44:12.445003 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 11:44:12.445154 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12832,"mutex_wait_us":29,"thread_start_us":6470,"threads_started":1}
I20260924 11:44:12.445777 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.421166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.0.97.2:33029)
0924 11:44:12.437126 (+ 15960us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.437132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.437153 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:44:12.437219 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.437224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.437603 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.437804 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:12.438629 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.440117 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.442360 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.443909 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.444041 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.444322 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.445215 (+ 893us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.445393 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15814,"thread_start_us":15725,"threads_started":1}
I20260924 11:44:12.445950 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.420984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33029 (local address 127.0.0.1:59894)
0924 11:44:12.430405 (+ 9421us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:12.430463 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:12.430665 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:12.438036 (+ 7371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:12.438040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:12.438062 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:12.438448 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.438458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.440263 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.440273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.442211 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.442226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.444669 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.444678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.444896 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.444920 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 11:44:12.445029 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 11:44:12.445190 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9081,"mutex_wait_us":44,"thread_start_us":13744,"threads_started":1}
I20260924 11:44:12.446427 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.424136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.0.97.1:33257)
0924 11:44:12.435881 (+ 11745us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.435893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.436591 (+ 698us) server_negotiation.cc:407] Connection header received
0924 11:44:12.436814 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.436824 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.437047 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.437236 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:12.438058 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.439326 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.441620 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.443129 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.443244 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.443454 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.445072 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.446188 (+ 1116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11498,"thread_start_us":11319,"threads_started":1}
I20260924 11:44:12.448364 16263 tablet_service.cc:1467] Processing CreateTablet for tablet 7feaa39794eb46bbb745437c901b434b (DEFAULT_TABLE table=test-table [id=35ab5bc5ae0b4e67a9a8126685d912a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:44:12.448858 16130 tablet_service.cc:1467] Processing CreateTablet for tablet 7feaa39794eb46bbb745437c901b434b (DEFAULT_TABLE table=test-table [id=35ab5bc5ae0b4e67a9a8126685d912a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:44:12.449901 16263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7feaa39794eb46bbb745437c901b434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:12.449967 16130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7feaa39794eb46bbb745437c901b434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:12.449636 15996 tablet_service.cc:1467] Processing CreateTablet for tablet 7feaa39794eb46bbb745437c901b434b (DEFAULT_TABLE table=test-table [id=35ab5bc5ae0b4e67a9a8126685d912a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:44:12.450893 15996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7feaa39794eb46bbb745437c901b434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:12.461218 16346 tablet_bootstrap.cc:492] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Bootstrap starting.
I20260924 11:44:12.463824 16347 tablet_bootstrap.cc:492] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Bootstrap starting.
I20260924 11:44:12.464632 16346 tablet_bootstrap.cc:654] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:12.465269 16348 tablet_bootstrap.cc:492] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Bootstrap starting.
I20260924 11:44:12.466049 16346 log.cc:824] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:12.467177 16347 tablet_bootstrap.cc:654] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:12.468389 16347 log.cc:824] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:12.468767 16346 tablet_bootstrap.cc:492] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: No bootstrap required, opened a new log
I20260924 11:44:12.469017 16346 ts_tablet_manager.cc:1397] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:44:12.469199 16348 tablet_bootstrap.cc:654] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:12.470690 16348 log.cc:824] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:12.473309 16348 tablet_bootstrap.cc:492] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: No bootstrap required, opened a new log
I20260924 11:44:12.473558 16348 ts_tablet_manager.cc:1397] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260924 11:44:12.473835 16347 tablet_bootstrap.cc:492] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: No bootstrap required, opened a new log
I20260924 11:44:12.474067 16347 ts_tablet_manager.cc:1397] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 11:44:12.479065 16346 raft_consensus.cc:348] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.479497 16346 raft_consensus.cc:374] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:12.479583 16346 raft_consensus.cc:729] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda79c7217e54f9e8bde533abf1a5e3a, State: Initialized, Role: FOLLOWER
I20260924 11:44:12.480203 16346 consensus_queue.cc:260] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.480340 16348 raft_consensus.cc:348] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.480815 16348 raft_consensus.cc:374] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:12.480931 16348 raft_consensus.cc:729] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14056ec8151847e88ec570fcd96ec5c0, State: Initialized, Role: FOLLOWER
I20260924 11:44:12.481487 16195 heartbeater.cc:503] Master 127.0.97.62:37105 was elected leader, sending a full tablet report...
I20260924 11:44:12.481552 16348 consensus_queue.cc:260] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.482050 16346 ts_tablet_manager.cc:1428] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 11:44:12.484246 16348 ts_tablet_manager.cc:1428] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260924 11:44:12.485735 16347 raft_consensus.cc:348] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.486131 16347 raft_consensus.cc:374] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:12.486233 16347 raft_consensus.cc:729] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4550cb304917423588a7db6e39d8d9f0, State: Initialized, Role: FOLLOWER
I20260924 11:44:12.486778 16347 consensus_queue.cc:260] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.487879 16328 heartbeater.cc:503] Master 127.0.97.62:37105 was elected leader, sending a full tablet report...
I20260924 11:44:12.488400 16347 ts_tablet_manager.cc:1428] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260924 11:44:12.584647 16352 raft_consensus.cc:482] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:44:12.584961 16352 raft_consensus.cc:504] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.586740 16352 leader_election.cc:288] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675), 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257)
I20260924 11:44:12.593206 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.587036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53019 (local address 127.0.97.3:35675)
0924 11:44:12.587237 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.587244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.588001 (+ 757us) server_negotiation.cc:407] Connection header received
0924 11:44:12.588250 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.588256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.588317 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.588422 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:12.588970 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.589673 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.591500 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.592323 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.592461 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.592566 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.592832 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.592942 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:44:12.593233 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.587210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35871 (local address 127.0.97.1:33257)
0924 11:44:12.587461 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.587470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.588163 (+ 693us) server_negotiation.cc:407] Connection header received
0924 11:44:12.588388 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.588392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.588448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.588540 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:12.589143 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.589813 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.591552 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.592377 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.592507 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.592608 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.592872 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.592981 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 11:44:12.593413 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.587144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.97.2:35871)
0924 11:44:12.588046 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:12.588060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:12.588217 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:12.588560 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:12.588564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:12.588601 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:12.589005 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.589013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.589957 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.589961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.591409 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.591420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.592577 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.592582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.592726 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.592745 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 11:44:12.592835 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 11:44:12.592939 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":46,"thread_start_us":573,"threads_started":1}
I20260924 11:44:12.593752 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.587061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35675 (local address 127.0.97.2:53019)
0924 11:44:12.587884 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:12.587899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:12.588068 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:12.588450 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:12.588455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:12.588478 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:12.588825 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.588833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.589794 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.589800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.591342 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:12.591353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.592531 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.592537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:12.592677 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.592700 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:44:12.592802 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:44:12.592915 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":38,"thread_start_us":401,"threads_started":1}
I20260924 11:44:12.594341 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feaa39794eb46bbb745437c901b434b" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14056ec8151847e88ec570fcd96ec5c0" is_pre_election: true
I20260924 11:44:12.594671 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feaa39794eb46bbb745437c901b434b" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4550cb304917423588a7db6e39d8d9f0" is_pre_election: true
I20260924 11:44:12.594887 16016 raft_consensus.cc:2454] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 0.
I20260924 11:44:12.595192 16283 raft_consensus.cc:2454] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 0.
I20260924 11:44:12.595597 16083 leader_election.cc:302] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14056ec8151847e88ec570fcd96ec5c0, eda79c7217e54f9e8bde533abf1a5e3a; no voters:
I20260924 11:44:12.595986 16352 raft_consensus.cc:2781] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:44:12.596123 16352 raft_consensus.cc:482] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:44:12.596266 16352 raft_consensus.cc:3037] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:12.598519 16352 raft_consensus.cc:504] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.599041 16352 leader_election.cc:288] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 election: Requested vote from peers 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675), 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257)
I20260924 11:44:12.599409 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feaa39794eb46bbb745437c901b434b" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4550cb304917423588a7db6e39d8d9f0"
I20260924 11:44:12.599483 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7feaa39794eb46bbb745437c901b434b" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14056ec8151847e88ec570fcd96ec5c0"
I20260924 11:44:12.599599 16283 raft_consensus.cc:3037] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:12.599668 16016 raft_consensus.cc:3037] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:12.602015 16016 raft_consensus.cc:2454] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 1.
I20260924 11:44:12.602025 16283 raft_consensus.cc:2454] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 1.
I20260924 11:44:12.602394 16083 leader_election.cc:302] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14056ec8151847e88ec570fcd96ec5c0, eda79c7217e54f9e8bde533abf1a5e3a; no voters:
I20260924 11:44:12.602710 16352 raft_consensus.cc:2781] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:44:12.603175 16352 raft_consensus.cc:686] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 1 LEADER]: Becoming Leader. State: Replica: eda79c7217e54f9e8bde533abf1a5e3a, State: Running, Role: LEADER
I20260924 11:44:12.603606 16352 consensus_queue.cc:237] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } }
I20260924 11:44:12.607744 15871 catalog_manager.cc:5495] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to eda79c7217e54f9e8bde533abf1a5e3a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } health_report { overall_health: UNKNOWN } } }
W20260924 11:44:12.609318 16329 tablet.cc:2369] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:44:12.624655 16062 tablet.cc:2369] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:44:12.631765 16196 tablet.cc:2369] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:44:12.667409 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.662225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.97.2:33029)
0924 11:44:12.662414 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.662418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.662447 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:44:12.662525 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.662528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.662604 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.662711 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:44:12.663030 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.663731 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.665243 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.665636 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.665726 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.666103 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:44:12.666615 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:44:12.666735 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.666947 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.667050 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:44:12.675750 16283 raft_consensus.cc:1261] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Refusing update from remote peer eda79c7217e54f9e8bde533abf1a5e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:44:12.675750 16016 raft_consensus.cc:1261] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Refusing update from remote peer eda79c7217e54f9e8bde533abf1a5e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:44:12.676818 16352 consensus_queue.cc:1035] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:44:12.677098 16357 consensus_queue.cc:1035] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:44:12.687423 16360 mvcc.cc:205] Tried to move back new op lower bound from 7332865034950135808 to 7332865034668306432. Current Snapshot: MvccSnapshot[applied={T|T < 7332865034950135808}]
I20260924 11:44:12.688181 16362 mvcc.cc:205] Tried to move back new op lower bound from 7332865034950135808 to 7332865034668306432. Current Snapshot: MvccSnapshot[applied={T|T < 7332865034950135808}]
I20260924 11:44:12.691570 16361 mvcc.cc:205] Tried to move back new op lower bound from 7332865034950135808 to 7332865034668306432. Current Snapshot: MvccSnapshot[applied={T|T < 7332865034950135808}]
I20260924 11:44:12.702641 15871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:44:12.702951 15871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:44:12.706166 15871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Sending DeleteTablet for 3 replicas of tablet 7feaa39794eb46bbb745437c901b434b
I20260924 11:44:12.707231 16263 tablet_service.cc:1514] Processing DeleteTablet for tablet 7feaa39794eb46bbb745437c901b434b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:44:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39172
I20260924 11:44:12.707302 15996 tablet_service.cc:1514] Processing DeleteTablet for tablet 7feaa39794eb46bbb745437c901b434b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:44:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51506
I20260924 11:44:12.707837 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 15933
I20260924 11:44:12.707916 16130 tablet_service.cc:1514] Processing DeleteTablet for tablet 7feaa39794eb46bbb745437c901b434b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:44:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59894
I20260924 11:44:12.720664 16372 tablet_replica.cc:330] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: stopping tablet replica
I20260924 11:44:12.721153 16372 raft_consensus.cc:2229] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:44:12.721973 16372 raft_consensus.cc:2258] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:44:12.724257 16372 ts_tablet_manager.cc:1905] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:12.726279 16371 tablet_replica.cc:330] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: stopping tablet replica
I20260924 11:44:12.726913 16370 tablet_replica.cc:330] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: stopping tablet replica
I20260924 11:44:12.727358 16370 raft_consensus.cc:2229] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:44:12.727838 16371 raft_consensus.cc:2229] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:44:12.728391 16370 raft_consensus.cc:2258] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:44:12.728871 16371 raft_consensus.cc:2258] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:44:12.731211 16371 ts_tablet_manager.cc:1905] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:12.731613 16372 ts_tablet_manager.cc:1918] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:12.731766 16372 log.cc:1196] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/7feaa39794eb46bbb745437c901b434b
I20260924 11:44:12.732370 16372 ts_tablet_manager.cc:1939] T 7feaa39794eb46bbb745437c901b434b P eda79c7217e54f9e8bde533abf1a5e3a: Deleting consensus metadata
I20260924 11:44:12.733108 16370 ts_tablet_manager.cc:1905] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:12.734175 15859 catalog_manager.cc:4841] TS eda79c7217e54f9e8bde533abf1a5e3a (127.0.97.2:33029): tablet 7feaa39794eb46bbb745437c901b434b (table test-table [id=35ab5bc5ae0b4e67a9a8126685d912a7]) successfully deleted
Sep 24 11:44:12 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:44:12.741057 16370 ts_tablet_manager.cc:1918] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:12.741372 16370 log.cc:1196] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/7feaa39794eb46bbb745437c901b434b
I20260924 11:44:12.741981 16370 ts_tablet_manager.cc:1939] T 7feaa39794eb46bbb745437c901b434b P 4550cb304917423588a7db6e39d8d9f0: Deleting consensus metadata
I20260924 11:44:12.741993 16371 ts_tablet_manager.cc:1918] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:12.742126 16371 log.cc:1196] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/7feaa39794eb46bbb745437c901b434b
I20260924 11:44:12.742609 16371 ts_tablet_manager.cc:1939] T 7feaa39794eb46bbb745437c901b434b P 14056ec8151847e88ec570fcd96ec5c0: Deleting consensus metadata
I20260924 11:44:12.743902 15858 catalog_manager.cc:4841] TS 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675): tablet 7feaa39794eb46bbb745437c901b434b (table test-table [id=35ab5bc5ae0b4e67a9a8126685d912a7]) successfully deleted
I20260924 11:44:12.744226 15856 catalog_manager.cc:4841] TS 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257): tablet 7feaa39794eb46bbb745437c901b434b (table test-table [id=35ab5bc5ae0b4e67a9a8126685d912a7]) successfully deleted
I20260924 11:44:12.745071 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.708906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.0.97.1:33257)
0924 11:44:12.709059 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.709064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.725456 (+ 16392us) server_negotiation.cc:407] Connection header received
0924 11:44:12.725726 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.725732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.725840 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.725951 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:12.727972 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.729150 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.731389 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.732019 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.732133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.737364 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:12.737375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:12.737391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:12.737425 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:12.741502 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.742367 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.742376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.742379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.742537 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.743046 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.743051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.743056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.743471 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:12.743734 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.744513 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.744704 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:44:12.745776 16056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:12.837889 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 15933
I20260924 11:44:12.855386 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16067
Sep 24 11:44:12 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:44:12.872303 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:12.856108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.0.97.2:33029)
0924 11:44:12.856309 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:12.856325 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:12.856472 (+ 147us) server_negotiation.cc:407] Connection header received
0924 11:44:12.856656 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:12.856659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:12.856721 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:12.856805 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:12.858556 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.859268 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.861287 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:12.861699 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:12.861786 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:12.866612 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:12.866622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:12.866635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:12.866665 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:12.869553 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.870174 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.870181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.870183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.870352 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:12.870729 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:12.870734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:12.870737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:12.871091 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:12.871416 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:12.871716 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 11:44:12.871908 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:44:12.872975 16190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:12.981417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16067
I20260924 11:44:12.999346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16200
Sep 24 11:44:13 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:44:13.014364 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:13.000097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.0.97.3:35675)
0924 11:44:13.000313 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:13.000318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:13.000358 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:44:13.000532 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:13.000536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:13.000613 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:13.000695 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:13.002174 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:13.002886 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:13.004189 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:13.004550 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:13.004639 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:13.008790 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:13.008802 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:13.008817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:13.008850 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:13.011453 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:13.012405 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:13.012416 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:13.012418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:13.012548 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:13.012904 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:13.012912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:13.012916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:13.013231 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:13.013515 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:13.013811 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:44:13.014012 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:44:13.015009 16323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:13.102372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16200
I20260924 11:44:13.119956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 15840
I20260924 11:44:13.121130 15902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:13.227056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 15840
I20260924 11:44:33.247377 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37105
--webserver_interface=127.0.97.62
--webserver_port=38751
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:44:33.573094 16378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:33.573411 16378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:33.573606 16378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:33.580746 16378 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:44:33.580869 16378 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:44:33.580968 16378 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:44:33.581072 16378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:44:33.581180 16378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:33.581285 16378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:33.581378 16378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:44:33.581462 16378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:44:33.591624 16378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:33.592808 16378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:33.594056 16378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:33.615447 16384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:33.615382 16385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:33.615423 16387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:33.616367 16378 server_base.cc:1034] running on GCE node
I20260924 11:44:33.617134 16378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:33.618551 16378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:33.619798 16378 hybrid_clock.cc:648] HybridClock initialized: now 1790250273619723 us; error 68 us; skew 500 ppm
I20260924 11:44:33.620276 16378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:33 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:33.623624 16378 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:44:33.625218 16378 webserver.cc:460] Webserver started at http://127.0.97.62:38751/ using document root <none> and password file <none>
I20260924 11:44:33.626000 16378 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:33.626142 16378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:33.631917 16378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:44:33.634665 16394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:33.635964 16378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:44:33.636205 16378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "93a29c0f4d654c70ba8eec99c0d0739a"
format_stamp: "Formatted at 2026-09-24 11:44:10 on dist-test-slave-qhrh"
server_key: "bfc5b5cd54cc88350049a3de639e7095"
server_key_iv: "cafb79c6cf4b5e1745025b15adc11478"
server_key_version: "encryptionkey@0"
I20260924 11:44:33.637255 16378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:33.660640 16378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:33.664520 16378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:33.664788 16378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:33.682566 16378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37105
I20260924 11:44:33.682596 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37105 every 8 connection(s)
I20260924 11:44:33.684026 16378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:44:33.690182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16378
I20260924 11:44:33.691100 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33257
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37105
--webserver_port=41701
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:44:33.691340 16446 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:44:33.696028 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap starting.
I20260924 11:44:33.715260 16446 log.cc:824] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:33.726435 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:44:33.726986 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap complete.
I20260924 11:44:33.735605 16446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:33.736442 16446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a29c0f4d654c70ba8eec99c0d0739a, State: Initialized, Role: FOLLOWER
I20260924 11:44:33.737094 16446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:33.737340 16446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:44:33.737460 16446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:44:33.737646 16446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:44:33.741460 16446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:33.741933 16446 leader_election.cc:302] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a29c0f4d654c70ba8eec99c0d0739a; no voters:
I20260924 11:44:33.742606 16446 leader_election.cc:288] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:44:33.742725 16450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:44:33.746738 16450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 LEADER]: Becoming Leader. State: Replica: 93a29c0f4d654c70ba8eec99c0d0739a, State: Running, Role: LEADER
I20260924 11:44:33.747323 16450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:33.747872 16446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:44:33.749637 16452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a29c0f4d654c70ba8eec99c0d0739a. Latest consensus state: current_term: 2 leader_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } } }
I20260924 11:44:33.749995 16452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: This master's current role is: LEADER
I20260924 11:44:33.750118 16451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } } }
I20260924 11:44:33.750614 16451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: This master's current role is: LEADER
I20260924 11:44:33.757022 16457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:44:33.765314 16457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:44:33.766613 16457 catalog_manager.cc:1259] Loaded cluster ID: a11a2939b90e4c6f96af3aed8f903b04
I20260924 11:44:33.766705 16457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:44:33.770483 16457 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:44:33.771797 16457 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Loaded TSK: 0
I20260924 11:44:33.814771 16457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Generated new TSK 1
I20260924 11:44:33.815335 16457 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:44:34.048800 16448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:34.049118 16448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:34.049252 16448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:34.056035 16448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:34.056195 16448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:34.056371 16448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:44:34.065634 16448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41701
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:34.066844 16448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:34.068136 16448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:34.082731 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:34.082769 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:34.083549 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:34.085333 16448 server_base.cc:1034] running on GCE node
I20260924 11:44:34.085959 16448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:34.087054 16448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:34.088230 16448 hybrid_clock.cc:648] HybridClock initialized: now 1790250274088182 us; error 39 us; skew 500 ppm
I20260924 11:44:34.088531 16448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:34 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:34.091536 16448 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:44:34.093118 16448 webserver.cc:460] Webserver started at http://127.0.97.1:41701/ using document root <none> and password file <none>
I20260924 11:44:34.093792 16448 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:34.093941 16448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:34.099167 16448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:44:34.101877 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.103129 16448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:44:34.103267 16448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "14056ec8151847e88ec570fcd96ec5c0"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "5b83435e39b917b546d53a564a59b465"
server_key_iv: "f0b2a13993e81bf229eb36caa010699c"
server_key_version: "encryptionkey@0"
I20260924 11:44:34.104130 16448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:34.135028 16448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:34.138868 16448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:34.139113 16448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:34.140009 16448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:34.141390 16448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:34.141470 16448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.141611 16448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:34.141700 16448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.173709 16448 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33257
I20260924 11:44:34.173794 16595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33257 every 8 connection(s)
I20260924 11:44:34.175063 16448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:44:34.175560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16448
I20260924 11:44:34.176344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33029
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37105
--webserver_port=40319
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:44:34 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:34.197956 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:34.178323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52919 (local address 127.0.97.62:37105)
0924 11:44:34.178912 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:34.178930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:34.179503 (+ 573us) server_negotiation.cc:407] Connection header received
0924 11:44:34.181655 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:34.181673 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:34.182099 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:34.182496 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:34.183754 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.185105 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.186922 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.187548 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.187759 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:34.190980 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:34.190987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:34.191000 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:34.191033 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:34.193824 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:34.194473 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:34.194480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:34.194485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:34.194628 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:34.195044 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:34.195050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:34.195053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:34.195466 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:34.195663 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:34.196406 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 11:44:34.196798 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260924 11:44:34.197963 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:34.178509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:52919)
0924 11:44:34.179336 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:34.179376 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:34.181111 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:34.182572 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:34.182579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:34.183055 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:44:34.183554 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:34.183571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.185266 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.185272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:34.186779 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:34.186790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.187792 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.187799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:34.188063 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:34.188777 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:44:34.188798 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:44:34.190730 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:44:34.193956 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:34.193964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:34.193976 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:34.194342 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:34.194736 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:34.194741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:34.194744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:34.194918 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:34.195609 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:44:34.195617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:44:34.195923 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 11:44:34.196194 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 11:44:34.196702 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":360,"threads_started":1}
I20260924 11:44:34.200853 16596 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37105
I20260924 11:44:34.201141 16596 heartbeater.cc:464] Registering TS with master...
I20260924 11:44:34.201802 16596 heartbeater.cc:511] Master 127.0.97.62:37105 requested a full tablet report, sending...
I20260924 11:44:34.203709 16411 ts_manager.cc:194] Registered new tserver with Master: 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257)
I20260924 11:44:34.205247 16411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52919
W20260924 11:44:34.488047 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:34.488380 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:34.488536 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:34.495169 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:34.495306 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:34.495477 16600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:44:34.504388 16600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40319
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:34.505520 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:34.506850 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:34.518846 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:34.518833 16606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:34.518879 16608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:34.519801 16600 server_base.cc:1034] running on GCE node
I20260924 11:44:34.520396 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:34.521212 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:34.522408 16600 hybrid_clock.cc:648] HybridClock initialized: now 1790250274522353 us; error 53 us; skew 500 ppm
I20260924 11:44:34.522696 16600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:34 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:34.525665 16600 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:44:34.527041 16600 webserver.cc:460] Webserver started at http://127.0.97.2:40319/ using document root <none> and password file <none>
I20260924 11:44:34.527679 16600 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:34.527853 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:34.532785 16600 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:44:34.535254 16615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.536463 16600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:44:34.536657 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "eda79c7217e54f9e8bde533abf1a5e3a"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "dacfeae78a3c18c3d10865d7dccf8ea6"
server_key_iv: "38999ea1922b27187835d16e495e5c2b"
server_key_version: "encryptionkey@0"
I20260924 11:44:34.537567 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:34.562676 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:34.566720 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:34.566977 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:34.567914 16600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:34.569156 16600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:34.569296 16600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.569416 16600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:34.569501 16600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.599936 16600 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33029
I20260924 11:44:34.600004 16727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33029 every 8 connection(s)
I20260924 11:44:34.601414 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:44:34.608899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16600
I20260924 11:44:34.609589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35675
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37105
--webserver_port=35591
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:44:34 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:34.620909 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:34.603803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43001 (local address 127.0.97.62:37105)
0924 11:44:34.603966 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:34.603970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:34.605060 (+ 1090us) server_negotiation.cc:407] Connection header received
0924 11:44:34.606878 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:34.606882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:34.606953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:34.607137 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:34.608857 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.609909 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.611415 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.611806 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.611951 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:34.615172 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:34.615177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:34.615180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:34.615222 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:34.617844 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:34.618720 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:34.618723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:34.618725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:34.618846 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:34.619296 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:34.619299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:34.619301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:34.619535 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:34.619640 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:34.620480 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 11:44:34.620640 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:44:34.622341 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:34.604198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:43001)
0924 11:44:34.604895 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:34.604936 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:34.606532 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:34.607362 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:34.607370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:34.607959 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:44:34.608681 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:34.608704 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.610056 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.610059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:34.611268 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:34.611278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:34.612013 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:34.612021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:34.612297 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:34.612949 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:44:34.612968 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:44:34.614942 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:44:34.617971 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:34.617982 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:34.617999 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:34.618602 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:34.618953 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:34.618960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:34.618967 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:34.619187 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:34.619668 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:44:34.619678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:44:34.620102 (+ 424us) client_negotiation.cc:769] Sending connection context
0924 11:44:34.620398 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 11:44:34.620943 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":273,"threads_started":1}
I20260924 11:44:34.623904 16728 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37105
I20260924 11:44:34.624245 16728 heartbeater.cc:464] Registering TS with master...
I20260924 11:44:34.624918 16728 heartbeater.cc:511] Master 127.0.97.62:37105 requested a full tablet report, sending...
I20260924 11:44:34.626319 16411 ts_manager.cc:194] Registered new tserver with Master: eda79c7217e54f9e8bde533abf1a5e3a (127.0.97.2:33029)
I20260924 11:44:34.627677 16411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43001
W20260924 11:44:34.915580 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:34.915805 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:34.915889 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:34.922967 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:34.923061 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:34.923170 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:44:34.932366 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35591
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:34.933462 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:34.934582 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:34.946905 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:34.946885 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:34.947351 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:34.947458 16731 server_base.cc:1034] running on GCE node
I20260924 11:44:34.948086 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:34.948953 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:34.950155 16731 hybrid_clock.cc:648] HybridClock initialized: now 1790250274950117 us; error 27 us; skew 500 ppm
I20260924 11:44:34.950448 16731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:34 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:34.953327 16731 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:44:34.954685 16731 webserver.cc:460] Webserver started at http://127.0.97.3:35591/ using document root <none> and password file <none>
I20260924 11:44:34.955327 16731 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:34.955466 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:34.960242 16731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:44:34.962754 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.963891 16731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:44:34.964051 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "4550cb304917423588a7db6e39d8d9f0"
format_stamp: "Formatted at 2026-09-24 11:44:12 on dist-test-slave-qhrh"
server_key: "d8bb14beb7b3e283ea6f5ec7c14a1047"
server_key_iv: "71877c2a70e037296b1935f88f44311a"
server_key_version: "encryptionkey@0"
I20260924 11:44:34.964939 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:34.987272 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:34.991159 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:34.991400 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:34.992322 16731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:34.993491 16731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:34.993613 16731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:34.993728 16731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:34.993820 16731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:35.028201 16731 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35675
I20260924 11:44:35.028292 16858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35675 every 8 connection(s)
I20260924 11:44:35.029577 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:44:35.033798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16731
Sep 24 11:44:35 dist-test-slave-qhrh krb5kdc[15827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:44:35.052043 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.034677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.0.97.62:37105)
0924 11:44:35.036357 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.036361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.036375 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:44:35.036415 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.036418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.036463 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.036544 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:35.038509 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.039462 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.041385 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.042040 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.042148 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.045380 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:35.045384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:35.045387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:35.045416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:35.049082 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.049875 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.049879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.049881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.049991 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.050403 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.050407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.050410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.050835 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:35.051172 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.051487 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.051716 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":1444,"threads_started":1}
I20260924 11:44:35.053172 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.032038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57577 (local address 127.0.97.62:37105)
0924 11:44:35.032277 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.032283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.033277 (+ 994us) server_negotiation.cc:407] Connection header received
0924 11:44:35.035606 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.035613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.035701 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.035808 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:35.038073 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.039052 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.041189 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.041878 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.042011 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.045918 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:35.045927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:35.045932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:35.045982 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:35.049516 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.050308 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.050311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.050313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.050458 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.051172 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.051178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.051183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.051564 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:35.051712 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.052599 (+ 887us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.052818 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:44:35.054111 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.032501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:57577)
0924 11:44:35.033134 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.033171 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:35.035340 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:35.036260 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:35.036271 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:35.037045 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:44:35.037885 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.037913 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.039196 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.039200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.041042 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.041057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.042067 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.042072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.042374 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.043110 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:44:35.043138 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:44:35.045688 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:44:35.049637 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:35.049647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:35.049665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:35.050194 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:35.050564 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:44:35.050567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:44:35.050569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:44:35.050969 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:44:35.051715 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:44:35.051724 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:44:35.052093 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 11:44:35.052448 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 11:44:35.052955 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":250,"threads_started":1}
I20260924 11:44:35.055610 16859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37105
I20260924 11:44:35.055863 16859 heartbeater.cc:464] Registering TS with master...
I20260924 11:44:35.056550 16859 heartbeater.cc:511] Master 127.0.97.62:37105 requested a full tablet report, sending...
I20260924 11:44:35.057874 16411 ts_manager.cc:194] Registered new tserver with Master: 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675)
I20260924 11:44:35.058588 16411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57577
I20260924 11:44:35.066653 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:44:35.072803 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.067499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.0.97.62:37105)
0924 11:44:35.067664 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.067671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.068103 (+ 432us) server_negotiation.cc:407] Connection header received
0924 11:44:35.068354 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.068357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.068499 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.068609 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:44:35.069015 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.069703 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.071142 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.071546 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.071644 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.072105 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:44:35.072364 (+ 259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 11:44:35.072627 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49052: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 11:44:35.072926 16598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49052: token expired
W20260924 11:44:35.073414 16334 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:37105)
I20260924 11:44:35.084837 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.073822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.97.62:37105)
0924 11:44:35.073969 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.073973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.074057 (+ 84us) server_negotiation.cc:407] Connection header received
0924 11:44:35.074169 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.074172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.074219 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.074288 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:35.075491 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.076111 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.077587 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.077962 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.078039 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.080373 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:35.080377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:35.080380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:35.080413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:35.082896 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.083452 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.083455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.083458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.083561 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.083867 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.083870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.083872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.084095 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:35.084311 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.084489 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.084621 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:44:35.088068 16411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:44:35.095013 16411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:44:35.133414 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.105830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35675 (local address 127.0.0.1:52146)
0924 11:44:35.121930 (+ 16100us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.121973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:35.122160 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:35.124518 (+ 2358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:35.124526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:35.124554 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:35.124997 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.125009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.128241 (+ 3232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.128246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.130632 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.130646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.132406 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.132411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.132615 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.132678 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 11:44:35.132789 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 11:44:35.132972 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15820,"mutex_wait_us":69,"thread_start_us":16522,"threads_started":1}
I20260924 11:44:35.133505 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.107221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.97.2:33029)
0924 11:44:35.123427 (+ 16206us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.123436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.125150 (+ 1714us) server_negotiation.cc:407] Connection header received
0924 11:44:35.125277 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.125289 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.125534 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.125715 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:35.126646 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.127877 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.131010 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.132191 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.132341 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.132549 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.133033 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.133173 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16060,"thread_start_us":15977,"threads_started":1}
I20260924 11:44:35.133505 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.107281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.97.1:33257)
0924 11:44:35.125180 (+ 17899us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.125196 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.125745 (+ 549us) server_negotiation.cc:407] Connection header received
0924 11:44:35.126055 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.126062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.126263 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.126458 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:35.127210 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.128685 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.130532 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.131992 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.132127 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.132574 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.132975 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.133141 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17754,"thread_start_us":17629,"threads_started":1}
I20260924 11:44:35.134018 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.106111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.0.97.3:35675)
0924 11:44:35.123895 (+ 17784us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.123902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.123926 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:44:35.123997 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.124002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.124248 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.126087 (+ 1839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:35.126378 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.128058 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.130782 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.132114 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.133065 (+ 951us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.133296 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.133373 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.133762 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17648,"thread_start_us":17527,"threads_started":1}
I20260924 11:44:35.134196 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.106586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33029 (local address 127.0.0.1:53612)
0924 11:44:35.124577 (+ 17991us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.124593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:35.124719 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:35.125760 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:35.125767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:35.125848 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:35.126367 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.126383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.128070 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.128074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.130881 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.130890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.132693 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.132697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.132831 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.132852 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:44:35.133677 (+ 825us) client_negotiation.cc:240] Negotiation successful
0924 11:44:35.133788 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17881,"mutex_wait_us":51,"spinlock_wait_cycles":3328,"thread_start_us":18776,"threads_started":1}
I20260924 11:44:35.134565 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.106590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38162)
0924 11:44:35.125648 (+ 19058us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.125667 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:35.125865 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:35.126699 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:35.126704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:35.126726 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:35.127043 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.127054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.128852 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.128858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.130385 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.130396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.132403 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.132410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.132766 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.132822 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 11:44:35.132895 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 11:44:35.132986 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18925,"mutex_wait_us":1137,"spinlock_wait_cycles":1280,"thread_start_us":16811,"threads_started":1}
I20260924 11:44:35.137820 16531 tablet_service.cc:1467] Processing CreateTablet for tablet 56517ea6e9a74cd888ae361659584e69 (DEFAULT_TABLE table=test-table [id=5384c977b64c41cdaab1f61c1c123260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:44:35.137838 16663 tablet_service.cc:1467] Processing CreateTablet for tablet 56517ea6e9a74cd888ae361659584e69 (DEFAULT_TABLE table=test-table [id=5384c977b64c41cdaab1f61c1c123260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:44:35.137845 16794 tablet_service.cc:1467] Processing CreateTablet for tablet 56517ea6e9a74cd888ae361659584e69 (DEFAULT_TABLE table=test-table [id=5384c977b64c41cdaab1f61c1c123260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:44:35.138967 16531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56517ea6e9a74cd888ae361659584e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:35.139070 16794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56517ea6e9a74cd888ae361659584e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:35.139070 16663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56517ea6e9a74cd888ae361659584e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:44:35.151441 16875 tablet_bootstrap.cc:492] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Bootstrap starting.
I20260924 11:44:35.154889 16875 tablet_bootstrap.cc:654] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:35.155022 16876 tablet_bootstrap.cc:492] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Bootstrap starting.
I20260924 11:44:35.156365 16875 log.cc:824] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:35.156867 16877 tablet_bootstrap.cc:492] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Bootstrap starting.
I20260924 11:44:35.158702 16876 tablet_bootstrap.cc:654] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:35.159408 16877 tablet_bootstrap.cc:654] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Neither blocks nor log segments found. Creating new log.
I20260924 11:44:35.160125 16876 log.cc:824] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:35.160429 16877 log.cc:824] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:35.163408 16876 tablet_bootstrap.cc:492] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: No bootstrap required, opened a new log
I20260924 11:44:35.163425 16877 tablet_bootstrap.cc:492] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: No bootstrap required, opened a new log
I20260924 11:44:35.163676 16877 ts_tablet_manager.cc:1397] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 11:44:35.163679 16875 tablet_bootstrap.cc:492] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: No bootstrap required, opened a new log
I20260924 11:44:35.163681 16876 ts_tablet_manager.cc:1397] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:44:35.163949 16875 ts_tablet_manager.cc:1397] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260924 11:44:35.170646 16877 raft_consensus.cc:348] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.170979 16877 raft_consensus.cc:374] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:35.171092 16877 raft_consensus.cc:729] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda79c7217e54f9e8bde533abf1a5e3a, State: Initialized, Role: FOLLOWER
I20260924 11:44:35.171682 16877 consensus_queue.cc:260] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.174182 16875 raft_consensus.cc:348] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.174625 16875 raft_consensus.cc:374] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:35.174713 16875 raft_consensus.cc:729] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14056ec8151847e88ec570fcd96ec5c0, State: Initialized, Role: FOLLOWER
I20260924 11:44:35.174496 16876 raft_consensus.cc:348] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.174855 16876 raft_consensus.cc:374] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:44:35.174932 16876 raft_consensus.cc:729] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4550cb304917423588a7db6e39d8d9f0, State: Initialized, Role: FOLLOWER
I20260924 11:44:35.175248 16875 consensus_queue.cc:260] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.175365 16876 consensus_queue.cc:260] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.183148 16859 heartbeater.cc:503] Master 127.0.97.62:37105 was elected leader, sending a full tablet report...
I20260924 11:44:35.183161 16875 ts_tablet_manager.cc:1428] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
I20260924 11:44:35.183362 16876 ts_tablet_manager.cc:1428] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20260924 11:44:35.183852 16728 heartbeater.cc:503] Master 127.0.97.62:37105 was elected leader, sending a full tablet report...
I20260924 11:44:35.185882 16877 ts_tablet_manager.cc:1428] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
I20260924 11:44:35.186393 16596 heartbeater.cc:503] Master 127.0.97.62:37105 was elected leader, sending a full tablet report...
I20260924 11:44:35.196923 16881 raft_consensus.cc:482] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:44:35.197160 16881 raft_consensus.cc:504] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.198676 16881 leader_election.cc:288] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675), 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257)
I20260924 11:44:35.205432 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.199069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48653 (local address 127.0.97.1:33257)
0924 11:44:35.199240 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.199248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.199990 (+ 742us) server_negotiation.cc:407] Connection header received
0924 11:44:35.200185 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.200193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.200260 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.200360 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:35.201018 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.201707 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.203622 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.204435 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.204581 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.204693 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.205042 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.205190 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:44:35.205526 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.198982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48491 (local address 127.0.97.3:35675)
0924 11:44:35.199170 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.199178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.199877 (+ 699us) server_negotiation.cc:407] Connection header received
0924 11:44:35.200103 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.200107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.200221 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.200311 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:44:35.200952 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.201657 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.203631 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.204457 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.204595 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.204761 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.205183 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.205309 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:44:35.205613 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.198976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35675 (local address 127.0.97.2:48491)
0924 11:44:35.199770 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.199785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:35.199964 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:35.200332 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:35.200336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:35.200360 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:35.200816 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.200826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.201870 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.201875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.203489 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.203500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.204841 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.204847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.205024 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.205053 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:44:35.205142 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 11:44:35.205250 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":138,"thread_start_us":450,"threads_started":1}
I20260924 11:44:35.206189 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.199064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.97.2:48653)
0924 11:44:35.199888 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.199902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:44:35.200013 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:44:35.200374 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:44:35.200378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:44:35.200401 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:44:35.200878 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.200887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.201892 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.201900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.203477 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:44:35.203488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.204654 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.204660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:44:35.204853 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.204886 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:44:35.205774 (+ 888us) client_negotiation.cc:240] Negotiation successful
0924 11:44:35.205892 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":32,"thread_start_us":460,"threads_started":1}
I20260924 11:44:35.206593 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ea6e9a74cd888ae361659584e69" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4550cb304917423588a7db6e39d8d9f0" is_pre_election: true
I20260924 11:44:35.207091 16814 raft_consensus.cc:2454] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 0.
I20260924 11:44:35.207116 16551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ea6e9a74cd888ae361659584e69" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14056ec8151847e88ec570fcd96ec5c0" is_pre_election: true
I20260924 11:44:35.207688 16618 leader_election.cc:302] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4550cb304917423588a7db6e39d8d9f0, eda79c7217e54f9e8bde533abf1a5e3a; no voters:
I20260924 11:44:35.207641 16551 raft_consensus.cc:2454] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 0.
I20260924 11:44:35.208062 16881 raft_consensus.cc:2781] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:44:35.208246 16881 raft_consensus.cc:482] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:44:35.208372 16881 raft_consensus.cc:3037] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:35.210405 16881 raft_consensus.cc:504] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.211036 16881 leader_election.cc:288] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 election: Requested vote from peers 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675), 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257)
I20260924 11:44:35.211397 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ea6e9a74cd888ae361659584e69" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4550cb304917423588a7db6e39d8d9f0"
I20260924 11:44:35.211480 16551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56517ea6e9a74cd888ae361659584e69" candidate_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14056ec8151847e88ec570fcd96ec5c0"
I20260924 11:44:35.211587 16814 raft_consensus.cc:3037] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:35.211717 16551 raft_consensus.cc:3037] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:44:35.213829 16814 raft_consensus.cc:2454] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 1.
I20260924 11:44:35.214159 16551 raft_consensus.cc:2454] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda79c7217e54f9e8bde533abf1a5e3a in term 1.
I20260924 11:44:35.214224 16618 leader_election.cc:302] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4550cb304917423588a7db6e39d8d9f0, eda79c7217e54f9e8bde533abf1a5e3a; no voters:
I20260924 11:44:35.214506 16881 raft_consensus.cc:2781] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:44:35.214963 16881 raft_consensus.cc:686] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 1 LEADER]: Becoming Leader. State: Replica: eda79c7217e54f9e8bde533abf1a5e3a, State: Running, Role: LEADER
I20260924 11:44:35.215400 16881 consensus_queue.cc:237] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } }
I20260924 11:44:35.219993 16409 catalog_manager.cc:5495] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to eda79c7217e54f9e8bde533abf1a5e3a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eda79c7217e54f9e8bde533abf1a5e3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33029 } health_report { overall_health: HEALTHY } } }
I20260924 11:44:35.235358 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.230629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.97.2:33029)
0924 11:44:35.230812 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.230816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.230832 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:44:35.230904 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.230908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.230981 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.231105 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:44:35.231429 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.232115 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.233459 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.233892 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.233980 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.234419 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:44:35.234891 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:44:35.234989 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.235089 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.235149 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:44:35.243391 16551 raft_consensus.cc:1261] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Refusing update from remote peer eda79c7217e54f9e8bde533abf1a5e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:44:35.243425 16814 raft_consensus.cc:1261] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Refusing update from remote peer eda79c7217e54f9e8bde533abf1a5e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:44:35.244302 16886 consensus_queue.cc:1035] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4550cb304917423588a7db6e39d8d9f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:44:35.244585 16881 consensus_queue.cc:1035] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14056ec8151847e88ec570fcd96ec5c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:44:35.272501 16410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:44:35.272770 16410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:44:35.276036 16410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Sending DeleteTablet for 3 replicas of tablet 56517ea6e9a74cd888ae361659584e69
I20260924 11:44:35.277187 16794 tablet_service.cc:1514] Processing DeleteTablet for tablet 56517ea6e9a74cd888ae361659584e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:44:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52146
I20260924 11:44:35.277185 16531 tablet_service.cc:1514] Processing DeleteTablet for tablet 56517ea6e9a74cd888ae361659584e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:44:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38162
I20260924 11:44:35.277420 16663 tablet_service.cc:1514] Processing DeleteTablet for tablet 56517ea6e9a74cd888ae361659584e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:44:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53612
I20260924 11:44:35.278448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16448
I20260924 11:44:35.292542 16900 tablet_replica.cc:330] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: stopping tablet replica
I20260924 11:44:35.293097 16900 raft_consensus.cc:2229] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:44:35.293715 16898 tablet_replica.cc:330] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: stopping tablet replica
I20260924 11:44:35.293915 16900 raft_consensus.cc:2258] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:44:35.294178 16898 raft_consensus.cc:2229] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:44:35.294970 16898 raft_consensus.cc:2258] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:44:35.296172 16900 ts_tablet_manager.cc:1905] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:35.297034 16898 ts_tablet_manager.cc:1905] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:35.301225 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.280291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.97.1:33257)
0924 11:44:35.280470 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.280476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.280496 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:44:35.280551 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.280558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.280662 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.280765 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:35.282678 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.283869 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.285937 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.286646 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.286765 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.291217 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:35.291234 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:35.291251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:35.291333 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:35.295955 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.297181 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.297190 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.297194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.297337 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.298405 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.298410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.298414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.298962 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:35.299251 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.300608 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.300835 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 11:44:35.303483 16900 ts_tablet_manager.cc:1918] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:35.303674 16900 log.cc:1196] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/56517ea6e9a74cd888ae361659584e69
I20260924 11:44:35.303865 16591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:35.304292 16900 ts_tablet_manager.cc:1939] T 56517ea6e9a74cd888ae361659584e69 P eda79c7217e54f9e8bde533abf1a5e3a: Deleting consensus metadata
I20260924 11:44:35.306396 16398 catalog_manager.cc:4841] TS eda79c7217e54f9e8bde533abf1a5e3a (127.0.97.2:33029): tablet 56517ea6e9a74cd888ae361659584e69 (table test-table [id=5384c977b64c41cdaab1f61c1c123260]) successfully deleted
I20260924 11:44:35.308902 16899 tablet_replica.cc:330] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: stopping tablet replica
I20260924 11:44:35.309279 16899 raft_consensus.cc:2229] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:44:35.309769 16899 raft_consensus.cc:2258] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:44:35.311335 16899 ts_tablet_manager.cc:1905] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:35.318838 16899 ts_tablet_manager.cc:1918] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:35.318984 16899 log.cc:1196] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/56517ea6e9a74cd888ae361659584e69
I20260924 11:44:35.319510 16899 ts_tablet_manager.cc:1939] T 56517ea6e9a74cd888ae361659584e69 P 4550cb304917423588a7db6e39d8d9f0: Deleting consensus metadata
I20260924 11:44:35.321439 16397 catalog_manager.cc:4841] TS 4550cb304917423588a7db6e39d8d9f0 (127.0.97.3:35675): tablet 56517ea6e9a74cd888ae361659584e69 (table test-table [id=5384c977b64c41cdaab1f61c1c123260]) successfully deleted
I20260924 11:44:35.401616 16898 ts_tablet_manager.cc:1918] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:35.401868 16898 log.cc:1196] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/56517ea6e9a74cd888ae361659584e69
I20260924 11:44:35.402348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16448
I20260924 11:44:35.402427 16898 ts_tablet_manager.cc:1939] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Deleting consensus metadata
W20260924 11:44:35.419771 16395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33257 (error 108)
I20260924 11:44:35.419926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16600
I20260924 11:44:35.421736 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.421365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38170)
0924 11:44:35.421528 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.421605 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:44:35.421974 16395 catalog_manager.cc:4584] TS 14056ec8151847e88ec570fcd96ec5c0 (127.0.97.1:33257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56517ea6e9a74cd888ae361659584e69: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
I20260924 11:44:35.436170 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.421392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.0.97.2:33029)
0924 11:44:35.421574 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.421581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.421601 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:44:35.421693 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.421699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.421818 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.421928 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:35.423949 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.424793 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.426703 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.427117 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.427209 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.430620 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:35.430632 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:35.430644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:35.430680 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:35.433497 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.434219 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.434225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.434226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.434379 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.434813 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.434816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.434818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.435154 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:35.435379 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.435743 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.435881 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:44:35.436882 16723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:35.475592 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.475101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38184)
0924 11:44:35.475314 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.475425 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 11:44:35.476544 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.476204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38186)
0924 11:44:35.476345 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.476427 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:44:35.537174 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.536596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38192)
0924 11:44:35.536876 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.536982 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 11:44:35.538239 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.537904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38198)
0924 11:44:35.538048 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.538119 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 11:44:35.545656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16600
I20260924 11:44:35.563709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16731
I20260924 11:44:35.580336 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.568337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.97.3:35675)
0924 11:44:35.568490 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:44:35.568496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:44:35.568556 (+ 60us) server_negotiation.cc:407] Connection header received
0924 11:44:35.568797 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:44:35.568802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:44:35.568878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:44:35.568997 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:44:35.570405 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.571065 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.572424 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:44:35.572780 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:44:35.572904 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:44:35.575176 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:44:35.575188 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:44:35.575201 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:44:35.575228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:44:35.577775 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.578392 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.578399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.578401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.578522 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:44:35.578880 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:44:35.578886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:44:35.578889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:44:35.579191 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:44:35.579469 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 11:44:35.579772 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:44:35.579985 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 11:44:35.580924 16854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:35.618829 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.618317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38202)
0924 11:44:35.618536 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.618636 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 11:44:35.619925 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:35.619448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33257 (local address 127.0.0.1:38208)
0924 11:44:35.619650 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:35.619761 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 11:44:35.665395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16731
I20260924 11:44:35.682126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16378
I20260924 11:44:35.683259 16441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:44:35.794490 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16378
I20260924 11:44:55.812480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37105
--webserver_interface=127.0.97.62
--webserver_port=38751
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:44:56.119891 16906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:56.120177 16906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:56.120273 16906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:56.127040 16906 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:44:56.127130 16906 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:44:56.127175 16906 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:44:56.127245 16906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:44:56.127297 16906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:56.127373 16906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:56.127417 16906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:44:56.127465 16906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:44:56.136273 16906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:56.137374 16906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:56.138554 16906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:56.149394 16911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:56.149428 16912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:56.149334 16914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:56.150197 16906 server_base.cc:1034] running on GCE node
I20260924 11:44:56.150821 16906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:56.152063 16906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:56.153272 16906 hybrid_clock.cc:648] HybridClock initialized: now 1790250296153192 us; error 56 us; skew 500 ppm
I20260924 11:44:56.153535 16906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:56 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:56.162992 16906 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:44:56.164503 16906 webserver.cc:460] Webserver started at http://127.0.97.62:38751/ using document root <none> and password file <none>
I20260924 11:44:56.165131 16906 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:56.165212 16906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:56.170210 16906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:44:56.172789 16921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:56.174041 16906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:44:56.174233 16906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "93a29c0f4d654c70ba8eec99c0d0739a"
format_stamp: "Formatted at 2026-09-24 11:44:10 on dist-test-slave-qhrh"
server_key: "bfc5b5cd54cc88350049a3de639e7095"
server_key_iv: "cafb79c6cf4b5e1745025b15adc11478"
server_key_version: "encryptionkey@0"
I20260924 11:44:56.175062 16906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:56.205021 16906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:56.208704 16906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:56.208928 16906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:56.226913 16906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37105
I20260924 11:44:56.226992 16972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37105 every 8 connection(s)
I20260924 11:44:56.228431 16906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:44:56.234786 16973 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:44:56.235394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16906
I20260924 11:44:56.236317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33257
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37105
--webserver_port=41701
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:44:56.239177 16973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap starting.
I20260924 11:44:56.259759 16973 log.cc:824] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:44:56.275246 16973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:44:56.275744 16973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Bootstrap complete.
I20260924 11:44:56.284934 16973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:56.285949 16973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a29c0f4d654c70ba8eec99c0d0739a, State: Initialized, Role: FOLLOWER
I20260924 11:44:56.286671 16973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:56.286890 16973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:44:56.287005 16973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:44:56.287195 16973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:44:56.290876 16973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:56.291347 16973 leader_election.cc:302] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a29c0f4d654c70ba8eec99c0d0739a; no voters:
I20260924 11:44:56.291985 16973 leader_election.cc:288] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 11:44:56.292111 16977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:44:56.293272 16977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [term 3 LEADER]: Becoming Leader. State: Replica: 93a29c0f4d654c70ba8eec99c0d0739a, State: Running, Role: LEADER
I20260924 11:44:56.294000 16977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } }
I20260924 11:44:56.294186 16973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:44:56.296350 16979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a29c0f4d654c70ba8eec99c0d0739a. Latest consensus state: current_term: 3 leader_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } } }
I20260924 11:44:56.296693 16979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: This master's current role is: LEADER
I20260924 11:44:56.297101 16978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a29c0f4d654c70ba8eec99c0d0739a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37105 } } }
I20260924 11:44:56.297343 16978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a [sys.catalog]: This master's current role is: LEADER
I20260924 11:44:56.307808 16984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:44:56.318801 16984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:44:56.320463 16984 catalog_manager.cc:1259] Loaded cluster ID: a11a2939b90e4c6f96af3aed8f903b04
I20260924 11:44:56.320639 16984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:44:56.324856 16984 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:44:56.326438 16984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Loaded TSK: 0
I20260924 11:44:56.326640 16984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93a29c0f4d654c70ba8eec99c0d0739a: Loaded TSK: 1
F20260924 11:44:56.327764 16984 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efd2d3049ad google::LogMessage::Fail() at ??:0
@ 0x7efd2d30788a google::LogMessage::SendToLog() at ??:0
@ 0x7efd2d3044de google::LogMessage::Flush() at ??:0
@ 0x7efd2d305a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efd310687d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efd3105773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efd42d331c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efd42d32783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efd42dbe2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260924 11:44:56.369217 16997 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7efd42dbe0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7efd42e085e9 std::function<>::operator()() at ??:0
@ 0x7efd42d36cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efd42d34c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efd42dbc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efd42dbc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efd2ff50345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efd2ff5391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efd2ff53799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efd2ff2eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efd332086db start_thread at ??:0
@ 0x7efd2c19e61f clone at ??:0
W20260924 11:44:56.621083 16975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:56.621333 16975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:56.621483 16975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:56.628551 16975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:56.628652 16975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:56.628859 16975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:44:56.638419 16975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41701
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:56.639655 16975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:56.640974 16975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:56.653455 17021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:56.653488 17023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:56.653488 17020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:56.654435 16975 server_base.cc:1034] running on GCE node
I20260924 11:44:56.654958 16975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:56.655810 16975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:56.657014 16975 hybrid_clock.cc:648] HybridClock initialized: now 1790250296656973 us; error 53 us; skew 500 ppm
I20260924 11:44:56.657303 16975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:56 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:56.660418 16975 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:44:56.661721 16975 webserver.cc:460] Webserver started at http://127.0.97.1:41701/ using document root <none> and password file <none>
I20260924 11:44:56.662371 16975 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:56.662513 16975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:56.667423 16975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:44:56.670058 17030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:56.671277 16975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:44:56.671425 16975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "14056ec8151847e88ec570fcd96ec5c0"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "5b83435e39b917b546d53a564a59b465"
server_key_iv: "f0b2a13993e81bf229eb36caa010699c"
server_key_version: "encryptionkey@0"
I20260924 11:44:56.672401 16975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:56.693879 16975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:56.698020 16975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:56.698283 16975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:56.699206 16975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:56.701000 17037 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 11:44:56.705884 17037 ts_tablet_manager.cc:1852] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 11:44:56.706812 17037 ts_tablet_manager.cc:1905] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:44:56.713456 17037 ts_tablet_manager.cc:1918] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:44:56.713680 17037 ts_tablet_manager.cc:1939] T 56517ea6e9a74cd888ae361659584e69 P 14056ec8151847e88ec570fcd96ec5c0: Deleting consensus metadata
I20260924 11:44:56.714272 16975 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 11:44:56.714396 16975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260924 11:44:56.714514 16975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:56.714620 16975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:56.746577 16975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33257
I20260924 11:44:56.746623 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33257 every 8 connection(s)
I20260924 11:44:56.747849 16975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:44:56.750820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 16975
I20260924 11:44:56.751422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33029
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37105
--webserver_port=40319
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:44:56.752729 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:56.750760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:38277)
0924 11:44:56.751626 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:56.751824 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
W20260924 11:44:56.753306 17144 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37105 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37105: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
I20260924 11:44:56.754130 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:56.753820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:39675)
0924 11:44:56.753948 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:56.754034 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 11:44:56.755003 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:56.754719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:39713)
0924 11:44:56.754829 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:56.754896 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 11:44:56.755263 17144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:44:57.059648 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:57.059929 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:57.060020 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:57.066959 17147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:57.067051 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:57.067162 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:44:57.076469 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40319
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:57.077591 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:57.078754 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:57.091162 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:57.091228 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:57.091670 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:57.092275 17147 server_base.cc:1034] running on GCE node
I20260924 11:44:57.092818 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:57.093617 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:57.094827 17147 hybrid_clock.cc:648] HybridClock initialized: now 1790250297094786 us; error 37 us; skew 500 ppm
I20260924 11:44:57.095152 17147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:57 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:57.098052 17147 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:44:57.099372 17147 webserver.cc:460] Webserver started at http://127.0.97.2:40319/ using document root <none> and password file <none>
I20260924 11:44:57.100039 17147 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:57.100188 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:57.104962 17147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:44:57.107340 17162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:57.108635 17147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:44:57.108760 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "eda79c7217e54f9e8bde533abf1a5e3a"
format_stamp: "Formatted at 2026-09-24 11:44:11 on dist-test-slave-qhrh"
server_key: "dacfeae78a3c18c3d10865d7dccf8ea6"
server_key_iv: "38999ea1922b27187835d16e495e5c2b"
server_key_version: "encryptionkey@0"
I20260924 11:44:57.109642 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:57.131856 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:57.135421 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:57.135614 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:57.136430 17147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:57.137589 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:57.137674 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:57.137741 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:57.137790 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:57.169520 17147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33029
I20260924 11:44:57.169557 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33029 every 8 connection(s)
I20260924 11:44:57.170794 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:44:57.173621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 17147
I20260924 11:44:57.174328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35675
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37105
--webserver_port=35591
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:44:57.175369 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.173582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:42093)
0924 11:44:57.174273 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.174488 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
W20260924 11:44:57.175920 17275 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37105 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37105: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
I20260924 11:44:57.176783 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.176471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:59829)
0924 11:44:57.176601 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.176660 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 11:44:57.177655 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.177274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:45179)
0924 11:44:57.177427 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.177524 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:44:57.177868 17275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:44:57.474274 17278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:44:57.474512 17278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:44:57.474593 17278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:44:57.481850 17278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:44:57.481938 17278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:44:57.482122 17278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:44:57.491137 17278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35591
--tserver_master_addrs=127.0.97.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:44:57.492317 17278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:44:57.493477 17278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:44:57.505460 17283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:57.505519 17284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:44:57.506032 17286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:44:57.506541 17278 server_base.cc:1034] running on GCE node
I20260924 11:44:57.507099 17278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:44:57.507916 17278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:44:57.509125 17278 hybrid_clock.cc:648] HybridClock initialized: now 1790250297509063 us; error 81 us; skew 500 ppm
I20260924 11:44:57.509420 17278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:44:57 dist-test-slave-qhrh krb5kdc[15827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:44:57.512317 17278 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:44:57.513746 17278 webserver.cc:460] Webserver started at http://127.0.97.3:35591/ using document root <none> and password file <none>
I20260924 11:44:57.514355 17278 fs_manager.cc:359] Metadata directory not provided
I20260924 11:44:57.514490 17278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:44:57.519505 17278 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:44:57.522042 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:57.523214 17278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:44:57.523372 17278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "4550cb304917423588a7db6e39d8d9f0"
format_stamp: "Formatted at 2026-09-24 11:44:12 on dist-test-slave-qhrh"
server_key: "d8bb14beb7b3e283ea6f5ec7c14a1047"
server_key_iv: "71877c2a70e037296b1935f88f44311a"
server_key_version: "encryptionkey@0"
I20260924 11:44:57.524314 17278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:44:57.545562 17278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:44:57.548946 17278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:44:57.549206 17278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:44:57.550119 17278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:44:57.551278 17278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:44:57.551405 17278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:57.551530 17278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:44:57.551632 17278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:44:57.582302 17278 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35675
I20260924 11:44:57.582376 17405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35675 every 8 connection(s)
I20260924 11:44:57.583596 17278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:44:57.587109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 17278
I20260924 11:44:57.588030 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.586318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:60379)
0924 11:44:57.587009 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.587250 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":323,"threads_started":1}
W20260924 11:44:57.588822 17406 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37105 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37105: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
I20260924 11:44:57.589525 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.589244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:37199)
0924 11:44:57.589363 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.589421 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 11:44:57.590188 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.589945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:42483)
0924 11:44:57.590039 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.590096 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 11:44:57.590410 17406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:44:57.756654 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:57.755890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:53499)
0924 11:44:57.756360 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:57.756452 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":201,"threads_started":1}
I20260924 11:44:58.179032 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:58.178316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:56453)
0924 11:44:58.178740 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:58.178818 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260924 11:44:58.591657 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:58.590908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:51249)
0924 11:44:58.591372 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:58.591448 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260924 11:44:58.758129 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:58.757466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:32991)
0924 11:44:58.757888 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:58.757968 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260924 11:44:59.180503 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:59.179812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:34557)
0924 11:44:59.180280 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:59.180351 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":206,"threads_started":1}
I20260924 11:44:59.593096 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:59.592510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:60989)
0924 11:44:59.592890 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:59.592955 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":195,"threads_started":1}
I20260924 11:44:59.759806 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:44:59.759102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:58741)
0924 11:44:59.759564 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 11:44:59.759643 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":231,"threads_started":1}
I20260924 11:45:00.181845 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:00.181269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:48937)
0924 11:45:00.181624 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:00.181690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":180,"threads_started":1}
I20260924 11:45:00.594429 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:00.593781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:44985)
0924 11:45:00.594171 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:00.594251 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":199,"threads_started":1}
I20260924 11:45:00.761298 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:00.760643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:54027)
0924 11:45:00.761047 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:00.761126 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260924 11:45:01.183326 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:01.182679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:52821)
0924 11:45:01.183061 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:01.183145 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":195,"threads_started":1}
I20260924 11:45:01.596022 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:01.595326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:40673)
0924 11:45:01.595775 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:01.595844 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":260,"threads_started":1}
I20260924 11:45:01.762872 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:01.762228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:50389)
0924 11:45:01.762646 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:01.762720 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":213,"threads_started":1}
I20260924 11:45:02.184965 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:02.184313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:37207)
0924 11:45:02.184727 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:02.184816 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260924 11:45:02.597421 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:02.596828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:52965)
0924 11:45:02.597214 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:02.597280 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":219,"threads_started":1}
I20260924 11:45:02.764475 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:02.763692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:55031)
0924 11:45:02.764113 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:02.764242 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":230,"threads_started":1}
I20260924 11:45:03.186610 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:03.185790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:49617)
0924 11:45:03.186353 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:03.186461 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":297,"threads_started":1}
I20260924 11:45:03.598925 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:03.598248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:44553)
0924 11:45:03.598650 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:03.598753 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":218,"threads_started":1}
I20260924 11:45:03.765983 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:03.765262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:48509)
0924 11:45:03.765707 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:03.765800 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":194,"threads_started":1}
I20260924 11:45:04.188285 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:04.187432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:57631)
0924 11:45:04.187961 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:04.188040 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
I20260924 11:45:04.600596 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:04.599796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:35905)
0924 11:45:04.600319 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:04.600388 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":254,"threads_started":1}
I20260924 11:45:04.767434 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:04.766791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:54797)
0924 11:45:04.767194 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:04.767266 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":217,"threads_started":1}
I20260924 11:45:05.190088 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:05.189124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:56667)
0924 11:45:05.189859 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:05.189932 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":463,"threads_started":1}
I20260924 11:45:05.602036 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:05.601424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:39059)
0924 11:45:05.601821 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:05.601893 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":215,"threads_started":1}
I20260924 11:45:05.768963 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:05.768308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:60109)
0924 11:45:05.768723 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:05.768823 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260924 11:45:06.191562 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:06.190898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:34357)
0924 11:45:06.191322 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:06.191391 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260924 11:45:06.604737 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:06.602843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:45955)
0924 11:45:06.604499 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:06.604576 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":1452,"threads_started":1}
I20260924 11:45:06.770519 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:06.769830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:57933)
0924 11:45:06.770230 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:06.770324 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":234,"threads_started":1}
I20260924 11:45:07.193096 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:07.192441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:45267)
0924 11:45:07.192857 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:07.192928 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":232,"threads_started":1}
I20260924 11:45:07.606199 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:07.605553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:51251)
0924 11:45:07.605969 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:07.606041 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":226,"threads_started":1}
I20260924 11:45:07.771958 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:07.771284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:45107)
0924 11:45:07.771721 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:07.771796 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260924 11:45:08.194723 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:08.194049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:52965)
0924 11:45:08.194449 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:08.194536 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
I20260924 11:45:08.607851 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:08.607206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:40141)
0924 11:45:08.607596 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:08.607689 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":187,"threads_started":1}
I20260924 11:45:08.773372 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:08.772742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:43567)
0924 11:45:08.773133 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:08.773202 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":212,"threads_started":1}
I20260924 11:45:09.196471 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:09.195765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:43175)
0924 11:45:09.196249 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:09.196328 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":245,"threads_started":1}
I20260924 11:45:09.609261 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:09.608629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:40133)
0924 11:45:09.609026 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:09.609103 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":206,"threads_started":1}
I20260924 11:45:09.774951 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:09.774323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:50975)
0924 11:45:09.774726 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:09.774800 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":197,"threads_started":1}
I20260924 11:45:10.197923 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:10.197267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:49961)
0924 11:45:10.197674 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:10.197743 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":194,"threads_started":1}
I20260924 11:45:10.610836 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:10.610145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:38859)
0924 11:45:10.610558 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:10.610635 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":199,"threads_started":1}
I20260924 11:45:10.776454 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:10.775716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:38209)
0924 11:45:10.776197 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:10.776272 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":234,"threads_started":1}
I20260924 11:45:11.199534 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:11.198908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:40851)
0924 11:45:11.199290 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:11.199373 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":206,"threads_started":1}
I20260924 11:45:11.612398 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:11.611679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:59961)
0924 11:45:11.612044 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:11.612114 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":195,"threads_started":1}
I20260924 11:45:11.778055 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:11.777404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.1:35611)
0924 11:45:11.777815 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:11.777897 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":194,"threads_started":1}
I20260924 11:45:12.201474 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:12.200786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.2:36655)
0924 11:45:12.201198 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:12.201290 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260924 11:45:12.591929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 16975
I20260924 11:45:12.608400 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:12.592859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.0.97.1:33257)
0924 11:45:12.593405 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:12.593419 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:12.593522 (+ 103us) server_negotiation.cc:407] Connection header received
0924 11:45:12.595288 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:12.595298 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:12.595700 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:12.595984 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:12.597259 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:12.598403 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:12.599579 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:12.600348 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:12.600480 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:12.602947 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:12.602953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:12.602966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:12.602999 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:12.605729 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:12.606310 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:12.606315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:12.606321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:12.606457 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:12.606734 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:12.606736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:12.606738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:12.607188 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:12.607397 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:12.607643 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 11:45:12.608051 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":281,"threads_started":1}
I20260924 11:45:12.609665 17139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:12.615090 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:12.613446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37105 (local address 127.0.97.3:50349)
0924 11:45:12.614784 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:12.614886 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":1058,"threads_started":1}
I20260924 11:45:12.694350 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 16975
I20260924 11:45:12.711690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 17147
I20260924 11:45:12.728583 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:12.712647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.0.97.2:33029)
0924 11:45:12.713175 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:12.713189 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:12.713225 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:45:12.714790 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:12.714799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:12.715201 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:12.715470 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:12.716775 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:12.717970 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:12.719230 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:12.719873 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:12.720005 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:12.722482 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:12.722489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:12.722502 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:12.722534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:12.725537 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:12.726140 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:12.726146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:12.726151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:12.726300 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:12.726617 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:12.726621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:12.726623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:12.727236 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:12.727457 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:12.727820 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 11:45:12.728307 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
I20260924 11:45:12.729939 17270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:12.808626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 17147
I20260924 11:45:12.825924 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 17278
I20260924 11:45:12.842268 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:12.826954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.0.97.3:35675)
0924 11:45:12.827563 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:12.827577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:12.827616 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:45:12.829289 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:12.829298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:12.829720 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:12.829993 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:12.831239 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:12.832511 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:12.833639 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:12.834263 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:12.834390 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:12.836719 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:12.836726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:12.836739 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:12.836772 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:12.839557 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:12.840117 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:12.840135 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:12.840192 (+ 57us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:12.840385 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:12.840701 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:12.840704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:12.840706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:12.841164 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:12.841386 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:12.841656 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 11:45:12.841923 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":315,"threads_started":1}
I20260924 11:45:12.843711 17401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:12.925426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 17278
2026-09-24T11:45:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63792 ms)
[----------] 30 tests from SecurityITest (285226 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:13 dist-test-slave-qhrh krb5kdc[17463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:13 dist-test-slave-qhrh krb5kdc[17463](info): set up 2 sockets
Sep 24 11:45:13 dist-test-slave-qhrh krb5kdc[17463](info): commencing operation
krb5kdc: starting...
W20260924 11:45:13.984110 388 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:14 dist-test-slave-qhrh krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:45:14.205381 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36097
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:14.513091 17476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:14.513384 17476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:14.513569 17476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:14.520272 17476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:14.520409 17476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:14.520489 17476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:14.520587 17476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:14.520653 17476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:14.529733 17476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:14.530927 17476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:14.532245 17476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:14.543642 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:14.543606 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:14.543678 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:14.544178 17476 server_base.cc:1034] running on GCE node
I20260924 11:45:14.544960 17476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:14.546181 17476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:14.547371 17476 hybrid_clock.cc:648] HybridClock initialized: now 1790250314547334 us; error 31 us; skew 500 ppm
I20260924 11:45:14.547710 17476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:14 dist-test-slave-qhrh krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:14.550617 17476 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:45:14.551965 17476 webserver.cc:460] Webserver started at http://127.0.97.62:39329/ using document root <none> and password file <none>
I20260924 11:45:14.552728 17476 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:14.552865 17476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:14.553166 17476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:14.555927 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "1facbf269b894465a1d25a300296533a"
format_stamp: "Formatted at 2026-09-24 11:45:14 on dist-test-slave-qhrh"
server_key: "7d044cdf3fa2309567561256ec8c04bd"
server_key_iv: "18f5e949fa14b6811e3b0c909b4b6925"
server_key_version: "encryptionkey@0"
I20260924 11:45:14.556746 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "1facbf269b894465a1d25a300296533a"
format_stamp: "Formatted at 2026-09-24 11:45:14 on dist-test-slave-qhrh"
server_key: "7d044cdf3fa2309567561256ec8c04bd"
server_key_iv: "18f5e949fa14b6811e3b0c909b4b6925"
server_key_version: "encryptionkey@0"
I20260924 11:45:14.561344 17476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:45:14.564553 17491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:14.565732 17476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:14.565899 17476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "1facbf269b894465a1d25a300296533a"
format_stamp: "Formatted at 2026-09-24 11:45:14 on dist-test-slave-qhrh"
server_key: "7d044cdf3fa2309567561256ec8c04bd"
server_key_iv: "18f5e949fa14b6811e3b0c909b4b6925"
server_key_version: "encryptionkey@0"
I20260924 11:45:14.566107 17476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:14.580224 17476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:14.583730 17476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:14.583983 17476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:14.601248 17476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36097
I20260924 11:45:14.601277 17542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36097 every 8 connection(s)
I20260924 11:45:14.602569 17476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:14.604799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 17476
I20260924 11:45:14.605002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:14.605772 388 external_mini_cluster.cc:1428] Setting key 572e66f515881abf4d7c387cc6a62e97
I20260924 11:45:14.606760 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:45:14 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:14.620004 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a: Bootstrap starting.
I20260924 11:45:14.623860 17543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:14.625114 17543 log.cc:824] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:14.628294 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a: No bootstrap required, opened a new log
I20260924 11:45:14.629279 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:14.607693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.97.62:36097)
0924 11:45:14.608421 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:14.608435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:14.608477 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:45:14.610193 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:14.610216 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:14.610723 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:14.611134 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:14.612541 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:14.614423 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:14.615821 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:14.616852 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:14.617029 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:14.621076 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:14.621086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:14.621104 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:14.621151 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:14.624975 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:14.625708 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:14.625717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:14.625724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:14.625898 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:14.626327 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:14.626353 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:14.626357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:14.626994 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:14.627258 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:14.627611 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 11:45:14.628005 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":274,"threads_started":1}
I20260924 11:45:14.635989 17543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facbf269b894465a1d25a300296533a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260924 11:45:14.636371 17543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:14.636444 17543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1facbf269b894465a1d25a300296533a, State: Initialized, Role: FOLLOWER
I20260924 11:45:14.636924 17543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facbf269b894465a1d25a300296533a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260924 11:45:14.637101 17543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:14.637203 17543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:14.637351 17543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:14.639469 17543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facbf269b894465a1d25a300296533a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260924 11:45:14.639863 17543 leader_election.cc:302] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1facbf269b894465a1d25a300296533a; no voters:
I20260924 11:45:14.640471 17543 leader_election.cc:288] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:14.640568 17548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:14.641244 17548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [term 1 LEADER]: Becoming Leader. State: Replica: 1facbf269b894465a1d25a300296533a, State: Running, Role: LEADER
I20260924 11:45:14.641763 17548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facbf269b894465a1d25a300296533a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260924 11:45:14.642570 17543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:14.644745 17550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1facbf269b894465a1d25a300296533a. Latest consensus state: current_term: 1 leader_uuid: "1facbf269b894465a1d25a300296533a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facbf269b894465a1d25a300296533a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } } }
I20260924 11:45:14.645113 17550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:14.645694 17549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1facbf269b894465a1d25a300296533a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facbf269b894465a1d25a300296533a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } } }
I20260924 11:45:14.645949 17549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:14.649147 17556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:14.654201 17556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:14.662843 17556 catalog_manager.cc:1347] Generated new cluster ID: 63a85c6e04f74172aba0472c943403b8
I20260924 11:45:14.662948 17556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:14.682655 17556 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:14.684459 17556 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:14.696271 17556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a: Generated new TSK 0
I20260924 11:45:14.697016 17556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:45:14.763343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36097
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:15.077984 17569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:15.078246 17569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:15.078328 17569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:15.085230 17569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:15.085325 17569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:15.085486 17569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:45:15.094406 17569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:15.095513 17569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:15.096909 17569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:15.109442 17575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:15.109491 17574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:15.109511 17577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:15.110296 17569 server_base.cc:1034] running on GCE node
I20260924 11:45:15.110839 17569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:15.111642 17569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:15.112843 17569 hybrid_clock.cc:648] HybridClock initialized: now 1790250315112802 us; error 39 us; skew 500 ppm
I20260924 11:45:15.113092 17569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:15 dist-test-slave-qhrh krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:15.115950 17569 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:45:15.117303 17569 webserver.cc:460] Webserver started at http://127.0.97.1:39569/ using document root <none> and password file <none>
I20260924 11:45:15.117939 17569 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:15.118072 17569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:15.118386 17569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:15.121151 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "4a08a73ed840486a87652bd669b632a1"
format_stamp: "Formatted at 2026-09-24 11:45:15 on dist-test-slave-qhrh"
server_key: "54bb8f35703f21ed322d7a120e181651"
server_key_iv: "fc1c50ffe341da72166f212ce919e49a"
server_key_version: "encryptionkey@0"
I20260924 11:45:15.121891 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4a08a73ed840486a87652bd669b632a1"
format_stamp: "Formatted at 2026-09-24 11:45:15 on dist-test-slave-qhrh"
server_key: "54bb8f35703f21ed322d7a120e181651"
server_key_iv: "fc1c50ffe341da72166f212ce919e49a"
server_key_version: "encryptionkey@0"
I20260924 11:45:15.126552 17569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:15.129784 17584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:15.130968 17569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:15.131134 17569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "4a08a73ed840486a87652bd669b632a1"
format_stamp: "Formatted at 2026-09-24 11:45:15 on dist-test-slave-qhrh"
server_key: "54bb8f35703f21ed322d7a120e181651"
server_key_iv: "fc1c50ffe341da72166f212ce919e49a"
server_key_version: "encryptionkey@0"
I20260924 11:45:15.131321 17569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:15.149634 17569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:15.153261 17569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:15.153515 17569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:15.154330 17569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:15.155539 17569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:15.155606 17569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:15.155763 17569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:15.155819 17569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:15.187309 17569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36941
I20260924 11:45:15.187393 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36941 every 8 connection(s)
I20260924 11:45:15.188556 17569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:45:15.197263 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 17569
I20260924 11:45:15.197468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:45:15.198267 388 external_mini_cluster.cc:1428] Setting key 7e91a51f5a150bc71807503824323c7b
Sep 24 11:45:15 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:15.212811 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:15.191003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53817 (local address 127.0.97.62:36097)
0924 11:45:15.191563 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:15.191572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:15.192989 (+ 1417us) server_negotiation.cc:407] Connection header received
0924 11:45:15.195634 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:15.195640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:15.195734 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:15.195885 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:15.198060 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.199114 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.201530 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.201917 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.202007 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:15.205741 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:15.205750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:15.205757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:15.205798 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:15.209163 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:15.210002 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:15.210009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:15.210015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:15.210262 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:15.211091 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:15.211096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:15.211099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:15.211362 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:15.211521 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:15.212273 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 11:45:15.212481 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":280,"threads_started":1}
I20260924 11:45:15.213632 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:15.191677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36097 (local address 127.0.97.1:53817)
0924 11:45:15.192815 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:15.192869 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:15.195294 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:15.196134 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:15.196166 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:15.197045 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:15.197867 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:15.197891 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.199354 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.199360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:15.201371 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:15.201388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.202140 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.202149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:15.202463 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:15.203109 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:15.203144 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:15.205523 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:15.209281 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:15.209291 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:15.209308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:15.209869 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:15.210496 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:15.210503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:15.210510 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:15.210916 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:15.211495 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:15.211503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:15.211859 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 11:45:15.212116 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 11:45:15.212594 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":540,"threads_started":1}
I20260924 11:45:15.215175 17697 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36097
I20260924 11:45:15.215467 17697 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:15.216104 17697 heartbeater.cc:511] Master 127.0.97.62:36097 requested a full tablet report, sending...
I20260924 11:45:15.218349 17508 ts_manager.cc:194] Registered new tserver with Master: 4a08a73ed840486a87652bd669b632a1 (127.0.97.1:36941)
I20260924 11:45:15.220562 17508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53817
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:45:15.253547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36097
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:15.568838 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:15.569116 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:15.569316 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:15.576534 17703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:15.576675 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:15.576856 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:45:15.586061 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:15.587249 17703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:15.588611 17703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:15.601583 17709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:15.601773 17708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:15.601661 17711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:15.602988 17703 server_base.cc:1034] running on GCE node
I20260924 11:45:15.603684 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:15.604808 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:15.606061 17703 hybrid_clock.cc:648] HybridClock initialized: now 1790250315605985 us; error 80 us; skew 500 ppm
I20260924 11:45:15.606436 17703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:15 dist-test-slave-qhrh krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:15.610020 17703 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:45:15.613377 17703 webserver.cc:460] Webserver started at http://127.0.97.2:33807/ using document root <none> and password file <none>
I20260924 11:45:15.614004 17703 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:15.614146 17703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:15.614429 17703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:15.617147 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "0ee69683e5744998a26eda90eaa8d36c"
format_stamp: "Formatted at 2026-09-24 11:45:15 on dist-test-slave-qhrh"
server_key: "39460407d8a1f7a5321ee09e36736547"
server_key_iv: "67d93e8557f83ad28cbcd1159f5da6c4"
server_key_version: "encryptionkey@0"
I20260924 11:45:15.617879 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0ee69683e5744998a26eda90eaa8d36c"
format_stamp: "Formatted at 2026-09-24 11:45:15 on dist-test-slave-qhrh"
server_key: "39460407d8a1f7a5321ee09e36736547"
server_key_iv: "67d93e8557f83ad28cbcd1159f5da6c4"
server_key_version: "encryptionkey@0"
I20260924 11:45:15.622677 17703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 11:45:15.625954 17718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:15.627140 17703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:15.627306 17703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "0ee69683e5744998a26eda90eaa8d36c"
format_stamp: "Formatted at 2026-09-24 11:45:15 on dist-test-slave-qhrh"
server_key: "39460407d8a1f7a5321ee09e36736547"
server_key_iv: "67d93e8557f83ad28cbcd1159f5da6c4"
server_key_version: "encryptionkey@0"
I20260924 11:45:15.627491 17703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:15.655669 17703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:15.659286 17703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:15.659551 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:15.660506 17703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:15.661741 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:15.661841 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:15.661984 17703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:15.662055 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:15.693240 17703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37013
I20260924 11:45:15.693282 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37013 every 8 connection(s)
I20260924 11:45:15.694392 17703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:45:15.694797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 17703
I20260924 11:45:15.694980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:45:15.695614 388 external_mini_cluster.cc:1428] Setting key 136c2e2df28bdd8f1834cab41c594f6d
Sep 24 11:45:15 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:15.713292 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:15.697140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51507 (local address 127.0.97.62:36097)
0924 11:45:15.697322 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:15.697327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:15.698642 (+ 1315us) server_negotiation.cc:407] Connection header received
0924 11:45:15.700458 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:15.700463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:15.700548 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:15.700664 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:15.702065 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.702775 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.704397 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.704780 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.704904 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:15.707999 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:15.708004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:15.708007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:15.708034 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:15.710410 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:15.711099 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:15.711107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:15.711112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:15.711237 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:15.711661 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:15.711668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:15.711676 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:15.711922 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:15.712073 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:15.712737 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 11:45:15.712960 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:45:15.714345 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:15.697775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36097 (local address 127.0.97.2:51507)
0924 11:45:15.698506 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:15.698539 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:15.700103 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:15.700882 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:15.700890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:15.701384 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:15.701904 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:15.701921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.702921 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.702931 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:15.704256 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:15.704271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:15.704989 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:15.704995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:15.705226 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:15.705889 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:15.705913 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:15.707798 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:15.710553 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:15.710561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:15.710576 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:15.710979 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:15.711347 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:15.711353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:15.711357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:15.711534 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:15.712044 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:15.712051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:15.712412 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 11:45:15.712667 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 11:45:15.713149 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":307,"threads_started":1}
I20260924 11:45:15.715787 17831 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36097
I20260924 11:45:15.716058 17831 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:15.716724 17831 heartbeater.cc:511] Master 127.0.97.62:36097 requested a full tablet report, sending...
I20260924 11:45:15.718212 17508 ts_manager.cc:194] Registered new tserver with Master: 0ee69683e5744998a26eda90eaa8d36c (127.0.97.2:37013)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 11:45:15.719053 17508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:45:15.747730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36097
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:16.057698 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:16.057979 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:16.058148 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:16.064981 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:16.065066 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:16.065220 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:45:16.074113 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:16.075301 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:16.076663 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:16.092646 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:16.092653 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:16.092669 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:16.093427 17836 server_base.cc:1034] running on GCE node
I20260924 11:45:16.094033 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:16.094880 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:16.096066 17836 hybrid_clock.cc:648] HybridClock initialized: now 1790250316096022 us; error 43 us; skew 500 ppm
I20260924 11:45:16.096391 17836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:16 dist-test-slave-qhrh krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:16.099542 17836 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:45:16.101001 17836 webserver.cc:460] Webserver started at http://127.0.97.3:35121/ using document root <none> and password file <none>
I20260924 11:45:16.101648 17836 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:16.101812 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:16.102103 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:16.104839 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "587ff68166e14154a4a9adfe3967904a"
format_stamp: "Formatted at 2026-09-24 11:45:16 on dist-test-slave-qhrh"
server_key: "8f62858a17a711f6d3edbe0bb417f594"
server_key_iv: "5abfdcba5afa9e3ddd8b2cfcf056af0a"
server_key_version: "encryptionkey@0"
I20260924 11:45:16.105568 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "587ff68166e14154a4a9adfe3967904a"
format_stamp: "Formatted at 2026-09-24 11:45:16 on dist-test-slave-qhrh"
server_key: "8f62858a17a711f6d3edbe0bb417f594"
server_key_iv: "5abfdcba5afa9e3ddd8b2cfcf056af0a"
server_key_version: "encryptionkey@0"
I20260924 11:45:16.110284 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:16.113523 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:16.114665 17836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:16.114845 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "587ff68166e14154a4a9adfe3967904a"
format_stamp: "Formatted at 2026-09-24 11:45:16 on dist-test-slave-qhrh"
server_key: "8f62858a17a711f6d3edbe0bb417f594"
server_key_iv: "5abfdcba5afa9e3ddd8b2cfcf056af0a"
server_key_version: "encryptionkey@0"
I20260924 11:45:16.115036 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:16.134456 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:16.138218 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:16.138478 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:16.139328 17836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:16.140641 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:16.140748 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:16.140892 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:16.140961 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:16.171640 17836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35159
I20260924 11:45:16.171742 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35159 every 8 connection(s)
I20260924 11:45:16.172838 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:45:16.178274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 17836
I20260924 11:45:16.178440 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:45:16.178922 388 external_mini_cluster.cc:1428] Setting key a548afa03d8d3bdcf9c794219e3ddfbe
Sep 24 11:45:16 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:16.190734 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.175102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60179 (local address 127.0.97.62:36097)
0924 11:45:16.175256 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.175260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.176328 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 11:45:16.178108 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.178113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.178193 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.178295 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:16.179734 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.180542 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.182060 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.182410 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.182541 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.185678 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:16.185682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:16.185685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:16.185714 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:16.188021 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.188670 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.188676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.188680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.188800 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.189196 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.189200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.189211 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.189433 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:16.189573 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.190168 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.190380 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:45:16.191586 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.175498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36097 (local address 127.0.97.3:60179)
0924 11:45:16.176130 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:16.176214 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:16.177828 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:16.178528 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:16.178567 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:16.179080 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:16.179561 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.179576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.180682 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.180688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.181915 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.181926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.182638 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.182646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.182869 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.183421 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:16.183444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:16.185480 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:16.188165 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:16.188173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:16.188187 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:16.188556 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:16.188902 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:16.188907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:16.188911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:16.189084 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:16.189547 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:16.189554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:16.189847 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 11:45:16.190066 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 11:45:16.190509 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260924 11:45:16.192970 17964 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36097
I20260924 11:45:16.193238 17964 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:16.193998 17964 heartbeater.cc:511] Master 127.0.97.62:36097 requested a full tablet report, sending...
I20260924 11:45:16.195266 17507 ts_manager.cc:194] Registered new tserver with Master: 587ff68166e14154a4a9adfe3967904a (127.0.97.3:35159)
I20260924 11:45:16.196071 17507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60179
I20260924 11:45:16.205674 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:45:16.221853 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.210064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.97.62:36097)
0924 11:45:16.210217 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.210221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.210691 (+ 470us) server_negotiation.cc:407] Connection header received
0924 11:45:16.210854 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.210858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.210913 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.210997 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:16.212360 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.213097 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.214296 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.214663 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.214779 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.216977 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:16.216981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:16.216984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:16.217011 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:16.219597 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.220252 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.220257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.220259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.220398 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.220741 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.220745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.220747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.221046 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:16.221222 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.221490 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.221630 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:45:16.223387 17697 heartbeater.cc:503] Master 127.0.97.62:36097 was elected leader, sending a full tablet report...
I20260924 11:45:16.225693 17507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:45:16.232424 17507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:45:16.254485 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.244331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.0.97.3:35159)
0924 11:45:16.247587 (+ 3256us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.247597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.247640 (+ 43us) server_negotiation.cc:407] Connection header received
0924 11:45:16.247733 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.247739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.247973 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.248802 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:16.249032 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.250166 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.252112 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.253434 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.253565 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.253778 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.254015 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.254171 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3084,"thread_start_us":2928,"threads_started":1}
I20260924 11:45:16.254528 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.243767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35159 (local address 127.0.0.1:34142)
0924 11:45:16.246931 (+ 3164us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:16.246948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:16.247044 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:16.248201 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:16.248204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:16.248229 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:16.248615 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.248626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.250287 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.250293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.251954 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.251965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.253597 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.253603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.253784 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.253831 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 11:45:16.253963 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:45:16.254127 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2980,"mutex_wait_us":49,"thread_start_us":303,"threads_started":1}
I20260924 11:45:16.254705 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.243715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36941 (local address 127.0.0.1:39094)
0924 11:45:16.244437 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:16.244475 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:16.244663 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:16.246188 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:16.246198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:16.246234 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:16.246791 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.246808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.249643 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.249647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.252040 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.252064 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.253898 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.253902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.254067 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.254095 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 11:45:16.254172 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 11:45:16.254304 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":87,"thread_start_us":2875,"threads_started":1}
I20260924 11:45:16.255385 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.244219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.0.97.1:36941)
0924 11:45:16.245454 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.245466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.245497 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:45:16.245598 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.245608 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.245954 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.246162 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:16.247079 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.249509 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.252252 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.253481 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.253578 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.253778 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.254949 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.255098 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":972,"threads_started":1}
I20260924 11:45:16.255851 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.247504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37013 (local address 127.0.0.1:60468)
0924 11:45:16.248261 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:16.248277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:16.248441 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:16.248911 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:16.248915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:16.248936 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:16.249241 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.249248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.251947 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.251952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.253245 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.253256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.255095 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.255101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.255272 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.255300 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 11:45:16.255376 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 11:45:16.255489 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":18,"thread_start_us":510,"threads_started":1}
I20260924 11:45:16.256474 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.247230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.0.97.2:37013)
0924 11:45:16.247856 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.247864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.248364 (+ 500us) server_negotiation.cc:407] Connection header received
0924 11:45:16.248600 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.248606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.248756 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.248925 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:16.249374 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.251759 (+ 2385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.253385 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.254674 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.254793 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.254967 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.256030 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.256197 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":384,"threads_started":1}
I20260924 11:45:16.258378 17899 tablet_service.cc:1467] Processing CreateTablet for tablet 7a69895204ab474cbdfdd5436df8be4d (DEFAULT_TABLE table=test-table [id=4ba4c1c8a1f7422587eecd5fca948ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:16.258513 17632 tablet_service.cc:1467] Processing CreateTablet for tablet 7a69895204ab474cbdfdd5436df8be4d (DEFAULT_TABLE table=test-table [id=4ba4c1c8a1f7422587eecd5fca948ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:16.259662 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a69895204ab474cbdfdd5436df8be4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:16.259815 17632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a69895204ab474cbdfdd5436df8be4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:16.259682 17766 tablet_service.cc:1467] Processing CreateTablet for tablet 7a69895204ab474cbdfdd5436df8be4d (DEFAULT_TABLE table=test-table [id=4ba4c1c8a1f7422587eecd5fca948ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:16.260928 17766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a69895204ab474cbdfdd5436df8be4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:16.270324 17982 tablet_bootstrap.cc:492] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Bootstrap starting.
I20260924 11:45:16.273617 17982 tablet_bootstrap.cc:654] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:16.274067 17984 tablet_bootstrap.cc:492] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Bootstrap starting.
I20260924 11:45:16.274928 17983 tablet_bootstrap.cc:492] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Bootstrap starting.
I20260924 11:45:16.275538 17982 log.cc:824] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:16.277129 17984 tablet_bootstrap.cc:654] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:16.277706 17983 tablet_bootstrap.cc:654] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:16.278299 17984 log.cc:824] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:16.278663 17983 log.cc:824] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:16.278685 17982 tablet_bootstrap.cc:492] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: No bootstrap required, opened a new log
I20260924 11:45:16.279018 17982 ts_tablet_manager.cc:1397] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260924 11:45:16.281350 17984 tablet_bootstrap.cc:492] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: No bootstrap required, opened a new log
I20260924 11:45:16.281613 17983 tablet_bootstrap.cc:492] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: No bootstrap required, opened a new log
I20260924 11:45:16.281620 17984 ts_tablet_manager.cc:1397] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:45:16.281852 17983 ts_tablet_manager.cc:1397] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 11:45:16.288568 17984 raft_consensus.cc:348] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.288942 17984 raft_consensus.cc:374] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:16.289083 17984 raft_consensus.cc:729] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee69683e5744998a26eda90eaa8d36c, State: Initialized, Role: FOLLOWER
I20260924 11:45:16.289441 17982 raft_consensus.cc:348] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.289803 17982 raft_consensus.cc:374] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:16.289677 17984 consensus_queue.cc:260] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.289888 17982 raft_consensus.cc:729] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587ff68166e14154a4a9adfe3967904a, State: Initialized, Role: FOLLOWER
I20260924 11:45:16.290512 17982 consensus_queue.cc:260] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.290747 17831 heartbeater.cc:503] Master 127.0.97.62:36097 was elected leader, sending a full tablet report...
I20260924 11:45:16.291357 17984 ts_tablet_manager.cc:1428] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 11:45:16.292049 17983 raft_consensus.cc:348] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.292444 17983 raft_consensus.cc:374] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:16.292539 17983 raft_consensus.cc:729] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a08a73ed840486a87652bd669b632a1, State: Initialized, Role: FOLLOWER
I20260924 11:45:16.292749 17982 ts_tablet_manager.cc:1428] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260924 11:45:16.293119 17983 consensus_queue.cc:260] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.293742 17964 heartbeater.cc:503] Master 127.0.97.62:36097 was elected leader, sending a full tablet report...
I20260924 11:45:16.297369 17983 ts_tablet_manager.cc:1428] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260924 11:45:16.424445 17965 tablet.cc:2369] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:45:16.440857 17698 tablet.cc:2369] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:45:16.446606 17832 tablet.cc:2369] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:16.454157 17989 raft_consensus.cc:482] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:16.454435 17989 raft_consensus.cc:504] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.456022 17989 leader_election.cc:288] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a08a73ed840486a87652bd669b632a1 (127.0.97.1:36941), 0ee69683e5744998a26eda90eaa8d36c (127.0.97.2:37013)
I20260924 11:45:16.462400 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.456544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36941 (local address 127.0.97.3:37691)
0924 11:45:16.456753 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:16.456770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:16.456874 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:16.457300 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:16.457305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:16.457337 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:16.457595 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.457607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.458792 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.458795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.460666 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.460680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.461910 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.461913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.462032 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.462054 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:45:16.462123 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 11:45:16.462211 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":56}
I20260924 11:45:16.462555 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.456546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37691 (local address 127.0.97.1:36941)
0924 11:45:16.456782 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.456788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.456814 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:45:16.457052 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.457059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.457141 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.457251 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:16.457763 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.458677 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.460813 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.461751 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.461900 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.461995 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.462180 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.462318 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:45:16.463502 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.457258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44781 (local address 127.0.97.2:37013)
0924 11:45:16.457573 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.457579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.457901 (+ 322us) server_negotiation.cc:407] Connection header received
0924 11:45:16.458018 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.458024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.458095 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.458188 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:16.458616 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.459332 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.461406 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.462619 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.462763 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.462851 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.463126 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.463243 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 11:45:16.463624 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.457018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37013 (local address 127.0.97.3:44781)
0924 11:45:16.457813 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:16.457826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:16.457912 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:16.458203 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:16.458205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:16.458222 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:16.458477 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.458483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.459481 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.459487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.461262 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:16.461272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.462857 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.462861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:16.462978 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.462998 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:45:16.463102 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 11:45:16.463221 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":1,"thread_start_us":469,"threads_started":1}
I20260924 11:45:16.463444 17652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a69895204ab474cbdfdd5436df8be4d" candidate_uuid: "587ff68166e14154a4a9adfe3967904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a08a73ed840486a87652bd669b632a1" is_pre_election: true
I20260924 11:45:16.464027 17652 raft_consensus.cc:2454] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587ff68166e14154a4a9adfe3967904a in term 0.
I20260924 11:45:16.464710 17854 leader_election.cc:302] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a08a73ed840486a87652bd669b632a1, 587ff68166e14154a4a9adfe3967904a; no voters:
I20260924 11:45:16.464605 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a69895204ab474cbdfdd5436df8be4d" candidate_uuid: "587ff68166e14154a4a9adfe3967904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee69683e5744998a26eda90eaa8d36c" is_pre_election: true
I20260924 11:45:16.465085 17989 raft_consensus.cc:2781] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:45:16.465129 17786 raft_consensus.cc:2454] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587ff68166e14154a4a9adfe3967904a in term 0.
I20260924 11:45:16.465227 17989 raft_consensus.cc:482] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:45:16.465332 17989 raft_consensus.cc:3037] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:16.467339 17989 raft_consensus.cc:504] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.467890 17989 leader_election.cc:288] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [CANDIDATE]: Term 1 election: Requested vote from peers 4a08a73ed840486a87652bd669b632a1 (127.0.97.1:36941), 0ee69683e5744998a26eda90eaa8d36c (127.0.97.2:37013)
I20260924 11:45:16.468427 17652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a69895204ab474cbdfdd5436df8be4d" candidate_uuid: "587ff68166e14154a4a9adfe3967904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a08a73ed840486a87652bd669b632a1"
I20260924 11:45:16.468500 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a69895204ab474cbdfdd5436df8be4d" candidate_uuid: "587ff68166e14154a4a9adfe3967904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee69683e5744998a26eda90eaa8d36c"
I20260924 11:45:16.468643 17652 raft_consensus.cc:3037] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:16.468700 17786 raft_consensus.cc:3037] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:16.471014 17786 raft_consensus.cc:2454] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587ff68166e14154a4a9adfe3967904a in term 1.
I20260924 11:45:16.471041 17652 raft_consensus.cc:2454] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587ff68166e14154a4a9adfe3967904a in term 1.
I20260924 11:45:16.471364 17854 leader_election.cc:302] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee69683e5744998a26eda90eaa8d36c, 587ff68166e14154a4a9adfe3967904a; no voters:
I20260924 11:45:16.471652 17989 raft_consensus.cc:2781] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:16.472103 17989 raft_consensus.cc:686] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 1 LEADER]: Becoming Leader. State: Replica: 587ff68166e14154a4a9adfe3967904a, State: Running, Role: LEADER
I20260924 11:45:16.472640 17989 consensus_queue.cc:237] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } }
I20260924 11:45:16.476681 17507 catalog_manager.cc:5495] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a reported cstate change: term changed from 0 to 1, leader changed from <none> to 587ff68166e14154a4a9adfe3967904a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "587ff68166e14154a4a9adfe3967904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "587ff68166e14154a4a9adfe3967904a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 } health_report { overall_health: UNKNOWN } } }
I20260924 11:45:16.547029 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.541287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.0.97.3:35159)
0924 11:45:16.541526 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.541531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.541635 (+ 104us) server_negotiation.cc:407] Connection header received
0924 11:45:16.541841 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.541844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.541938 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.542050 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:45:16.542402 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.543048 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.544640 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.545102 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.545208 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.545671 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:45:16.546325 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:45:16.546471 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.546649 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.546752 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:45:16.554841 17786 raft_consensus.cc:1261] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 1 FOLLOWER]: Refusing update from remote peer 587ff68166e14154a4a9adfe3967904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:16.554842 17652 raft_consensus.cc:1261] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 1 FOLLOWER]: Refusing update from remote peer 587ff68166e14154a4a9adfe3967904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:16.555752 17992 consensus_queue.cc:1035] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a08a73ed840486a87652bd669b632a1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:16.556083 17989 consensus_queue.cc:1035] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee69683e5744998a26eda90eaa8d36c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:16.580802 17508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:16.581063 17508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:16.584414 17508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1facbf269b894465a1d25a300296533a: Sending DeleteTablet for 3 replicas of tablet 7a69895204ab474cbdfdd5436df8be4d
I20260924 11:45:16.585386 17632 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a69895204ab474cbdfdd5436df8be4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39094
I20260924 11:45:16.585490 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a69895204ab474cbdfdd5436df8be4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34142
I20260924 11:45:16.586440 17766 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a69895204ab474cbdfdd5436df8be4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60468
I20260924 11:45:16.586740 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 17569
I20260924 11:45:16.598249 18005 tablet_replica.cc:330] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: stopping tablet replica
I20260924 11:45:16.598683 18005 raft_consensus.cc:2229] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:16.599238 18005 raft_consensus.cc:2258] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:16.600346 18006 tablet_replica.cc:330] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: stopping tablet replica
I20260924 11:45:16.600813 18006 raft_consensus.cc:2229] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:45:16.601526 18006 raft_consensus.cc:2258] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:16.603832 18005 ts_tablet_manager.cc:1905] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:16.609925 18007 tablet_replica.cc:330] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: stopping tablet replica
I20260924 11:45:16.610380 18007 raft_consensus.cc:2229] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:16.610891 18005 ts_tablet_manager.cc:1918] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:16.610913 18007 raft_consensus.cc:2258] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:16.611052 18005 log.cc:1196] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/7a69895204ab474cbdfdd5436df8be4d
I20260924 11:45:16.611577 18005 ts_tablet_manager.cc:1939] T 7a69895204ab474cbdfdd5436df8be4d P 4a08a73ed840486a87652bd669b632a1: Deleting consensus metadata
I20260924 11:45:16.613843 18006 ts_tablet_manager.cc:1905] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:16.614868 18007 ts_tablet_manager.cc:1905] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:45:16 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:45:16.620946 18006 ts_tablet_manager.cc:1918] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:16.621109 18006 log.cc:1196] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/7a69895204ab474cbdfdd5436df8be4d
I20260924 11:45:16.621613 18006 ts_tablet_manager.cc:1939] T 7a69895204ab474cbdfdd5436df8be4d P 587ff68166e14154a4a9adfe3967904a: Deleting consensus metadata
I20260924 11:45:16.623368 17495 catalog_manager.cc:4841] TS 587ff68166e14154a4a9adfe3967904a (127.0.97.3:35159): tablet 7a69895204ab474cbdfdd5436df8be4d (table test-table [id=4ba4c1c8a1f7422587eecd5fca948ae4]) successfully deleted
I20260924 11:45:16.624855 17494 catalog_manager.cc:4841] TS 4a08a73ed840486a87652bd669b632a1 (127.0.97.1:36941): tablet 7a69895204ab474cbdfdd5436df8be4d (table test-table [id=4ba4c1c8a1f7422587eecd5fca948ae4]) successfully deleted
I20260924 11:45:16.628062 18007 ts_tablet_manager.cc:1918] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:16.628245 18007 log.cc:1196] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/7a69895204ab474cbdfdd5436df8be4d
I20260924 11:45:16.628696 18007 ts_tablet_manager.cc:1939] T 7a69895204ab474cbdfdd5436df8be4d P 0ee69683e5744998a26eda90eaa8d36c: Deleting consensus metadata
I20260924 11:45:16.630312 17494 catalog_manager.cc:4841] TS 0ee69683e5744998a26eda90eaa8d36c (127.0.97.2:37013): tablet 7a69895204ab474cbdfdd5436df8be4d (table test-table [id=4ba4c1c8a1f7422587eecd5fca948ae4]) successfully deleted
I20260924 11:45:16.631583 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.588132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.0.97.1:36941)
0924 11:45:16.588375 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.588382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.602130 (+ 13748us) server_negotiation.cc:407] Connection header received
0924 11:45:16.602842 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.602847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.602927 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.602998 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:16.606286 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.607368 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.613047 (+ 5679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.613617 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.613701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.624298 (+ 10597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:16.624312 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:16.624328 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:16.624365 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:16.628125 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.629111 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.629120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.629124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.629298 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.629747 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.629754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.629758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.630212 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:16.630533 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.630951 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.631189 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:45:16.632397 17692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:16.718364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 17569
I20260924 11:45:16.736689 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 17703
Sep 24 11:45:16 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:45:16.752821 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.737460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.0.97.2:37013)
0924 11:45:16.737627 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.737630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.737701 (+ 71us) server_negotiation.cc:407] Connection header received
0924 11:45:16.737862 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.737865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.737924 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.737996 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:16.739544 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.740419 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.742312 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.742641 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.742759 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.747448 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:16.747476 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:16.747490 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:16.747535 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:16.750125 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.750828 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.750835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.750838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.750958 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.751311 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.751319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.751322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.751681 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:16.752000 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.752300 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.752505 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:45:16.753377 17826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:16.837855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 17703
I20260924 11:45:16.854776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 17836
Sep 24 11:45:16 dist-test-slave-qhrh krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:45:16.870302 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:16.855558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.0.97.3:35159)
0924 11:45:16.855744 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:16.855750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:16.855824 (+ 74us) server_negotiation.cc:407] Connection header received
0924 11:45:16.856007 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:16.856012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:16.856086 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:16.856223 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:16.857403 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.858252 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.859518 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:16.859983 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:16.860126 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:16.864482 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:16.864492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:16.864506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:16.864535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:16.867483 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.868126 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.868164 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.868169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.868351 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:16.868712 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:16.868717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:16.868721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:16.869077 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:16.869447 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:16.869714 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 11:45:16.869967 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:45:16.871058 17959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:16.986011 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 17836
I20260924 11:45:17.004357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 17476
I20260924 11:45:17.005488 17538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:17.113340 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 17476
2026-09-24T11:45:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4189 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T11:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:17Z Disabled control of system clock
I20260924 11:45:17.185045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39185
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:45:17.497545 18016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:17.497804 18016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:17.497920 18016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:17.504741 18016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:17.504848 18016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:17.504904 18016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:17.504993 18016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:17.505065 18016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:17.513727 18016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:17.514920 18016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:17.516728 18016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:17.528367 18024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:17.528402 18022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:17.528434 18021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:17.529095 18016 server_base.cc:1034] running on GCE node
I20260924 11:45:17.529872 18016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:17.531116 18016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:17.532342 18016 hybrid_clock.cc:648] HybridClock initialized: now 1790250317532314 us; error 47 us; skew 500 ppm
I20260924 11:45:17.532668 18016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:17.543978 18016 webserver.cc:460] Webserver started at http://127.0.97.62:40645/ using document root <none> and password file <none>
I20260924 11:45:17.544624 18016 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:17.544737 18016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:17.545046 18016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:17.547602 18016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "e6cdf922ead449c28c88bafce557c461"
format_stamp: "Formatted at 2026-09-24 11:45:17 on dist-test-slave-qhrh"
server_key: "e344954bcc4cc8e33f7c292e67acda02"
server_key_iv: "608367258cbfb88c279267547c80cb7e"
server_key_version: "encryptionkey@0"
I20260924 11:45:17.548395 18016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "e6cdf922ead449c28c88bafce557c461"
format_stamp: "Formatted at 2026-09-24 11:45:17 on dist-test-slave-qhrh"
server_key: "e344954bcc4cc8e33f7c292e67acda02"
server_key_iv: "608367258cbfb88c279267547c80cb7e"
server_key_version: "encryptionkey@0"
I20260924 11:45:17.553098 18016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:45:17.556303 18030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:17.557791 18016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:45:17.557996 18016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "e6cdf922ead449c28c88bafce557c461"
format_stamp: "Formatted at 2026-09-24 11:45:17 on dist-test-slave-qhrh"
server_key: "e344954bcc4cc8e33f7c292e67acda02"
server_key_iv: "608367258cbfb88c279267547c80cb7e"
server_key_version: "encryptionkey@0"
I20260924 11:45:17.558223 18016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:17.587419 18016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:17.588281 18016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:17.588562 18016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:17.605511 18016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39185
I20260924 11:45:17.605541 18081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39185 every 8 connection(s)
I20260924 11:45:17.606735 18016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:17.610452 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:17.615229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18016
I20260924 11:45:17.615437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:17.615916 388 external_mini_cluster.cc:1428] Setting key c96ebf61e666e2c9155603044d86f028
I20260924 11:45:17.621275 18082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461: Bootstrap starting.
I20260924 11:45:17.624250 18082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:17.625227 18082 log.cc:824] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:17.627830 18082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461: No bootstrap required, opened a new log
I20260924 11:45:17.628733 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:17.617781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.0.97.62:39185)
0924 11:45:17.618731 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:17.618742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:17.618793 (+ 51us) server_negotiation.cc:407] Connection header received
0924 11:45:17.620493 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:17.620514 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:17.620944 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:17.621308 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:17.621835 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:17.623113 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:17.624614 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:17.625253 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:17.625438 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:17.626533 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:17.626540 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:17.626551 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:17.627032 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:17.627099 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:17.627117 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:17.627234 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:17.627479 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 11:45:17.627763 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":331,"threads_started":1}
I20260924 11:45:17.635285 18082 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdf922ead449c28c88bafce557c461" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39185 } }
I20260924 11:45:17.635653 18082 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:17.635751 18082 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cdf922ead449c28c88bafce557c461, State: Initialized, Role: FOLLOWER
I20260924 11:45:17.636303 18082 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdf922ead449c28c88bafce557c461" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39185 } }
I20260924 11:45:17.636500 18082 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:17.636590 18082 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:17.636703 18082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:17.638897 18082 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdf922ead449c28c88bafce557c461" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39185 } }
I20260924 11:45:17.639302 18082 leader_election.cc:302] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6cdf922ead449c28c88bafce557c461; no voters:
I20260924 11:45:17.639802 18082 leader_election.cc:288] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:17.639894 18087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:17.640686 18087 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [term 1 LEADER]: Becoming Leader. State: Replica: e6cdf922ead449c28c88bafce557c461, State: Running, Role: LEADER
I20260924 11:45:17.641247 18087 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdf922ead449c28c88bafce557c461" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39185 } }
I20260924 11:45:17.641939 18082 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:17.643776 18089 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6cdf922ead449c28c88bafce557c461. Latest consensus state: current_term: 1 leader_uuid: "e6cdf922ead449c28c88bafce557c461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdf922ead449c28c88bafce557c461" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39185 } } }
I20260924 11:45:17.644233 18089 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:17.644105 18088 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6cdf922ead449c28c88bafce557c461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdf922ead449c28c88bafce557c461" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39185 } } }
I20260924 11:45:17.644495 18088 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:17.646682 18095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:17.650748 18095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:17.658623 18095 catalog_manager.cc:1347] Generated new cluster ID: 5f68921976e2404191486fabcd845704
I20260924 11:45:17.658711 18095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:17.679642 18095 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:17.680917 18095 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:17.688413 18095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461: Generated new TSK 0
I20260924 11:45:17.689143 18095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:17.698712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39185
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:45:18.013176 18106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:18.013445 18106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:18.013597 18106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:18.020260 18106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:18.020375 18106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:18.020552 18106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:45:18.028944 18106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:18.030131 18106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:18.031824 18106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:18.043910 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:18.044020 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:18.044291 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:18.045106 18106 server_base.cc:1034] running on GCE node
I20260924 11:45:18.045693 18106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:18.046607 18106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:18.047835 18106 hybrid_clock.cc:648] HybridClock initialized: now 1790250318047788 us; error 36 us; skew 500 ppm
I20260924 11:45:18.048130 18106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:18.050215 18106 webserver.cc:460] Webserver started at http://127.0.97.1:39819/ using document root <none> and password file <none>
I20260924 11:45:18.050944 18106 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:18.051064 18106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:18.051384 18106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:18.054229 18106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "45391230093649e7823830b767c2e7d0"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "39c50f7629c6cb1bf3d4158efd86d23a"
server_key_iv: "c9467ea83ca3f62ab24071ca074ea3c8"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.055007 18106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "45391230093649e7823830b767c2e7d0"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "39c50f7629c6cb1bf3d4158efd86d23a"
server_key_iv: "c9467ea83ca3f62ab24071ca074ea3c8"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.059962 18106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 11:45:18.063498 18120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.064723 18106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:18.064889 18106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "45391230093649e7823830b767c2e7d0"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "39c50f7629c6cb1bf3d4158efd86d23a"
server_key_iv: "c9467ea83ca3f62ab24071ca074ea3c8"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.065112 18106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:18.080322 18106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:18.081102 18106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:18.081373 18106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:18.082274 18106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:18.083912 18106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:18.084060 18106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.084205 18106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:18.084306 18106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.115738 18106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45661
I20260924 11:45:18.115823 18232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45661 every 8 connection(s)
I20260924 11:45:18.117030 18106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:45:18.125646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18106
I20260924 11:45:18.125808 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:45:18.126335 388 external_mini_cluster.cc:1428] Setting key 13ef255c03ece131d9fe3fa4d7acf810
I20260924 11:45:18.129297 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.119494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55375 (local address 127.0.97.62:39185)
0924 11:45:18.119720 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:18.119727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:18.121015 (+ 1288us) server_negotiation.cc:407] Connection header received
0924 11:45:18.122691 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:18.122695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:18.122758 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:18.122842 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:18.123761 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.124503 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.126321 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.126809 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.126956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.128545 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:18.128549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:18.128552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:18.128568 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:18.128597 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:18.128608 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:18.128701 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:18.128918 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 11:45:18.129006 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 11:45:18.129438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39185
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:45:18.130360 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.120012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39185 (local address 127.0.97.1:55375)
0924 11:45:18.120865 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:18.120898 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:18.122469 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:18.123025 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:18.123032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:18.123108 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:18.123625 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:18.123640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.124627 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.124630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:18.126186 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:18.126200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.127043 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.127050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:18.127293 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.128215 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:18.128218 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:18.128266 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:18.128359 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:18.128700 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:18.128713 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:18.128723 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:45:18.128898 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 11:45:18.129246 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":267,"threads_started":1}
I20260924 11:45:18.131883 18233 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39185
I20260924 11:45:18.132238 18233 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:18.132906 18233 heartbeater.cc:511] Master 127.0.97.62:39185 requested a full tablet report, sending...
I20260924 11:45:18.134716 18047 ts_manager.cc:194] Registered new tserver with Master: 45391230093649e7823830b767c2e7d0 (127.0.97.1:45661)
I20260924 11:45:18.136117 18047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:55375
W20260924 11:45:18.436714 18236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:18.436951 18236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:18.437088 18236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:18.443784 18236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:18.443867 18236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:18.444043 18236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:45:18.452723 18236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:18.453897 18236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:18.455611 18236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:18.468047 18244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:18.468127 18241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:18.468091 18242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:18.468887 18236 server_base.cc:1034] running on GCE node
I20260924 11:45:18.469517 18236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:18.470443 18236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:18.471674 18236 hybrid_clock.cc:648] HybridClock initialized: now 1790250318471631 us; error 33 us; skew 500 ppm
I20260924 11:45:18.471993 18236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:18.474149 18236 webserver.cc:460] Webserver started at http://127.0.97.2:40211/ using document root <none> and password file <none>
I20260924 11:45:18.474835 18236 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:18.474960 18236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:18.475261 18236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:18.477912 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "2a901f43d8974b8ca008226529faa87f"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "36c28f1becdbe4d3749582e3993799e2"
server_key_iv: "02b2bedb172f9bd4b8f6b48433806525"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.478578 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2a901f43d8974b8ca008226529faa87f"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "36c28f1becdbe4d3749582e3993799e2"
server_key_iv: "02b2bedb172f9bd4b8f6b48433806525"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.483419 18236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 11:45:18.486652 18250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.487741 18236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:18.487921 18236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "2a901f43d8974b8ca008226529faa87f"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "36c28f1becdbe4d3749582e3993799e2"
server_key_iv: "02b2bedb172f9bd4b8f6b48433806525"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.488116 18236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:18.508323 18236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:18.509058 18236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:18.509327 18236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:18.510212 18236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:18.511631 18236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:18.511751 18236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.511889 18236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:18.511976 18236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.543325 18236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33389
I20260924 11:45:18.543373 18362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33389 every 8 connection(s)
I20260924 11:45:18.544749 18236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:45:18.545123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18236
I20260924 11:45:18.545296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:45:18.545815 388 external_mini_cluster.cc:1428] Setting key 1ce8a531c6f1cef95ebfa8c9b31db3c8
I20260924 11:45:18.548818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39185
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:45:18.559649 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.547394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50721 (local address 127.0.97.62:39185)
0924 11:45:18.547629 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:18.547635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:18.549135 (+ 1500us) server_negotiation.cc:407] Connection header received
0924 11:45:18.551901 (+ 2766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:18.551905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:18.551970 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:18.552052 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:18.553480 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.554460 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.556861 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.557459 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.557610 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.558986 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:18.558991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:18.558994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:18.559009 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:18.559039 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:18.559049 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:18.559110 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:18.559357 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 11:45:18.559429 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 11:45:18.561095 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.547984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39185 (local address 127.0.97.2:50721)
0924 11:45:18.549001 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:18.549050 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:18.551611 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:18.552360 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:18.552374 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:18.552498 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:18.553291 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:18.553317 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.554633 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.554641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:18.556718 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:18.556734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.557704 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.557711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:18.557940 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.558630 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:18.558636 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:18.558686 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:18.558785 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:18.559132 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:18.559143 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:18.559152 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 11:45:18.559324 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 11:45:18.559714 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":314,"threads_started":1}
I20260924 11:45:18.562409 18363 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39185
I20260924 11:45:18.562717 18363 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:18.563503 18363 heartbeater.cc:511] Master 127.0.97.62:39185 requested a full tablet report, sending...
I20260924 11:45:18.564808 18047 ts_manager.cc:194] Registered new tserver with Master: 2a901f43d8974b8ca008226529faa87f (127.0.97.2:33389)
I20260924 11:45:18.565588 18047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50721
W20260924 11:45:18.856869 18366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:18.857151 18366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:18.857303 18366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:18.864540 18366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:18.864657 18366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:18.864810 18366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:45:18.873344 18366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:18.874452 18366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:18.876047 18366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:18.888078 18374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:18.888172 18372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:18.888593 18371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:18.888849 18366 server_base.cc:1034] running on GCE node
I20260924 11:45:18.889358 18366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:18.890164 18366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:18.891397 18366 hybrid_clock.cc:648] HybridClock initialized: now 1790250318891341 us; error 57 us; skew 500 ppm
I20260924 11:45:18.891741 18366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:18.893998 18366 webserver.cc:460] Webserver started at http://127.0.97.3:45693/ using document root <none> and password file <none>
I20260924 11:45:18.894629 18366 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:18.894737 18366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:18.895025 18366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:18.897583 18366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "13a0688cd4724fc8bc0290fb6496aed6"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "41b89b423f0bae4ac4d65253d5b898c2"
server_key_iv: "58bc1c1e55ab3e38bfb52899bc9532fd"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.898427 18366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "13a0688cd4724fc8bc0290fb6496aed6"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "41b89b423f0bae4ac4d65253d5b898c2"
server_key_iv: "58bc1c1e55ab3e38bfb52899bc9532fd"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.903796 18366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:45:18.906946 18380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.908051 18366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:18.908226 18366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "13a0688cd4724fc8bc0290fb6496aed6"
format_stamp: "Formatted at 2026-09-24 11:45:18 on dist-test-slave-qhrh"
server_key: "41b89b423f0bae4ac4d65253d5b898c2"
server_key_iv: "58bc1c1e55ab3e38bfb52899bc9532fd"
server_key_version: "encryptionkey@0"
I20260924 11:45:18.908421 18366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:18.920058 18366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:18.920792 18366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:18.921056 18366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:18.921892 18366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:18.923166 18366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:18.923280 18366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.923410 18366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:18.923493 18366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:18.954061 18366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36963
I20260924 11:45:18.954142 18492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36963 every 8 connection(s)
I20260924 11:45:18.955314 18366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:45:18.964870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18366
I20260924 11:45:18.965073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:45:18.965617 388 external_mini_cluster.cc:1428] Setting key 6b92b16815218460eefc7879ff92b2e8
I20260924 11:45:18.966863 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.957668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37331 (local address 127.0.97.62:39185)
0924 11:45:18.957840 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:18.957845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:18.959031 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 11:45:18.960754 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:18.960758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:18.960830 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:18.960979 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:18.961983 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.962728 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.964172 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.964600 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.964741 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.966080 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:18.966085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:18.966088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:18.966104 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:18.966133 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:18.966144 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:18.966213 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:18.966471 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 11:45:18.966549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:45:18.968065 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.958078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39185 (local address 127.0.97.3:37331)
0924 11:45:18.958897 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:18.958929 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:18.960495 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:18.961190 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:18.961197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:18.961277 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:18.961793 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:18.961808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.962874 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.962886 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:18.964013 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:18.964023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.964791 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.964796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:18.965062 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.965804 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:18.965807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:18.965840 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:18.965910 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:18.966241 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:18.966252 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:18.966260 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:45:18.966448 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 11:45:18.966828 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260924 11:45:18.969233 18493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39185
I20260924 11:45:18.969549 18493 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:18.970188 18493 heartbeater.cc:511] Master 127.0.97.62:39185 requested a full tablet report, sending...
I20260924 11:45:18.971393 18047 ts_manager.cc:194] Registered new tserver with Master: 13a0688cd4724fc8bc0290fb6496aed6 (127.0.97.3:36963)
I20260924 11:45:18.972213 18047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37331
I20260924 11:45:18.980752 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:45:18.989279 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:18.984355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.0.97.62:39185)
0924 11:45:18.984550 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:18.984555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:18.984796 (+ 241us) server_negotiation.cc:407] Connection header received
0924 11:45:18.984911 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:18.984914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:18.984998 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:18.985067 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:18.985450 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.986099 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.987321 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:18.987663 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:18.987742 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:18.988666 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:18.988670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:18.988673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:18.988687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:18.988715 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:18.988725 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:18.988791 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:18.988906 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 11:45:18.988969 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:18.992763 18047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:45:18.999100 18047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:45:19.022440 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.010450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45661 (local address 127.0.0.1:60886)
0924 11:45:19.015466 (+ 5016us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:19.015505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:19.015686 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:19.016677 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:19.016685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:19.016716 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:19.017135 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.017153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.018430 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.018438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.019977 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.019988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.021696 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.021704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.021946 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.022008 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 11:45:19.022091 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 11:45:19.022223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4775,"threads_started":1}
I20260924 11:45:19.024330 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.011035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.97.1:45661)
0924 11:45:19.015343 (+ 4308us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.015350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.015607 (+ 257us) server_negotiation.cc:407] Connection header received
0924 11:45:19.015983 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.015990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.016248 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.016421 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:19.017331 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.018294 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.020195 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.021466 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.023522 (+ 2056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.023767 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.023833 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.023990 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4153,"thread_start_us":4748,"threads_started":1}
I20260924 11:45:19.028976 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.012160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36963 (local address 127.0.0.1:43386)
0924 11:45:19.022667 (+ 10507us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:19.022682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:19.022765 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:19.023231 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:19.023235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:19.023255 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:19.023413 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.023422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.025043 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.025046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.026380 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.026395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.028025 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.028029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.028310 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.028466 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:45:19.028551 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 11:45:19.028667 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10391,"mutex_wait_us":112,"thread_start_us":8143,"threads_started":1}
I20260924 11:45:19.029182 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.011452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.0.97.2:33389)
0924 11:45:19.014969 (+ 3517us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.014979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.020708 (+ 5729us) server_negotiation.cc:407] Connection header received
0924 11:45:19.020862 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.020881 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.021163 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.021396 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:19.022028 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.023862 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.026581 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.027495 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.027569 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.027757 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.028624 (+ 867us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.028781 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":871,"threads_started":1}
I20260924 11:45:19.029206 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.012559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.0.97.3:36963)
0924 11:45:19.014941 (+ 2382us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.014951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.022863 (+ 7912us) server_negotiation.cc:407] Connection header received
0924 11:45:19.022935 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.022939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.023078 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.023242 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:19.023574 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.024897 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.026570 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.027875 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.028004 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.028270 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.028604 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.028753 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2176,"thread_start_us":1999,"threads_started":1}
I20260924 11:45:19.029322 18168 tablet_service.cc:1467] Processing CreateTablet for tablet ad6baf16786c44bc846ed200bacb7bdb (DEFAULT_TABLE table=test-table [id=a15d542a601e4f4ab7a2ec354fbdce41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:19.029923 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.011760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33389 (local address 127.0.0.1:34114)
0924 11:45:19.020559 (+ 8799us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:19.020578 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:19.020711 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:19.021398 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:19.021403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:19.021429 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:19.021845 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.021856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.024061 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.024069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.026367 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.026382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.028017 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.028023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.028312 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.028464 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 11:45:19.029564 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0924 11:45:19.029649 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8668,"mutex_wait_us":125,"thread_start_us":3348,"threads_started":1}
I20260924 11:45:19.032393 18428 tablet_service.cc:1467] Processing CreateTablet for tablet ad6baf16786c44bc846ed200bacb7bdb (DEFAULT_TABLE table=test-table [id=a15d542a601e4f4ab7a2ec354fbdce41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:19.032589 18298 tablet_service.cc:1467] Processing CreateTablet for tablet ad6baf16786c44bc846ed200bacb7bdb (DEFAULT_TABLE table=test-table [id=a15d542a601e4f4ab7a2ec354fbdce41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:19.033512 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6baf16786c44bc846ed200bacb7bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:19.033687 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6baf16786c44bc846ed200bacb7bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:19.038424 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6baf16786c44bc846ed200bacb7bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:19.044345 18511 tablet_bootstrap.cc:492] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Bootstrap starting.
I20260924 11:45:19.046187 18512 tablet_bootstrap.cc:492] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Bootstrap starting.
I20260924 11:45:19.047992 18511 tablet_bootstrap.cc:654] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:19.049444 18511 log.cc:824] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:19.049527 18512 tablet_bootstrap.cc:654] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:19.050503 18512 log.cc:824] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:19.052350 18514 tablet_bootstrap.cc:492] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Bootstrap starting.
I20260924 11:45:19.052506 18511 tablet_bootstrap.cc:492] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: No bootstrap required, opened a new log
I20260924 11:45:19.052788 18511 ts_tablet_manager.cc:1397] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 11:45:19.053336 18512 tablet_bootstrap.cc:492] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: No bootstrap required, opened a new log
I20260924 11:45:19.053562 18512 ts_tablet_manager.cc:1397] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:45:19.055060 18514 tablet_bootstrap.cc:654] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:19.056241 18514 log.cc:824] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:19.058605 18514 tablet_bootstrap.cc:492] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: No bootstrap required, opened a new log
I20260924 11:45:19.058840 18514 ts_tablet_manager.cc:1397] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 11:45:19.063268 18511 raft_consensus.cc:348] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.063586 18512 raft_consensus.cc:348] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.063876 18511 raft_consensus.cc:374] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:19.063977 18511 raft_consensus.cc:729] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a901f43d8974b8ca008226529faa87f, State: Initialized, Role: FOLLOWER
I20260924 11:45:19.064070 18512 raft_consensus.cc:374] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:19.064246 18512 raft_consensus.cc:729] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a0688cd4724fc8bc0290fb6496aed6, State: Initialized, Role: FOLLOWER
I20260924 11:45:19.064731 18511 consensus_queue.cc:260] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.064936 18512 consensus_queue.cc:260] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.065793 18363 heartbeater.cc:503] Master 127.0.97.62:39185 was elected leader, sending a full tablet report...
I20260924 11:45:19.066437 18511 ts_tablet_manager.cc:1428] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260924 11:45:19.066757 18514 raft_consensus.cc:348] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.067103 18514 raft_consensus.cc:374] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:19.067265 18514 raft_consensus.cc:729] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45391230093649e7823830b767c2e7d0, State: Initialized, Role: FOLLOWER
I20260924 11:45:19.067802 18514 consensus_queue.cc:260] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.072840 18493 heartbeater.cc:503] Master 127.0.97.62:39185 was elected leader, sending a full tablet report...
I20260924 11:45:19.073411 18512 ts_tablet_manager.cc:1428] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260924 11:45:19.074478 18233 heartbeater.cc:503] Master 127.0.97.62:39185 was elected leader, sending a full tablet report...
I20260924 11:45:19.074892 18514 ts_tablet_manager.cc:1428] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260924 11:45:19.108193 18518 raft_consensus.cc:482] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:19.108471 18518 raft_consensus.cc:504] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.110029 18518 leader_election.cc:288] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45391230093649e7823830b767c2e7d0 (127.0.97.1:45661), 2a901f43d8974b8ca008226529faa87f (127.0.97.2:33389)
I20260924 11:45:19.116022 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.110433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46827 (local address 127.0.97.2:33389)
0924 11:45:19.110615 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.110621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.110898 (+ 277us) server_negotiation.cc:407] Connection header received
0924 11:45:19.111126 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.111133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.111214 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.111323 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:19.111786 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.112686 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.114324 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.115115 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.115263 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.115366 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.115648 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.115783 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:45:19.116058 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.110406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33389 (local address 127.0.97.3:46827)
0924 11:45:19.110776 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:19.110802 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:19.110947 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:19.111409 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:19.111415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:19.111443 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:19.111622 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.111636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.112813 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.112820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.114158 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.114168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.115353 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.115359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.115497 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.115520 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:45:19.115621 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:45:19.115762 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":20}
I20260924 11:45:19.117089 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.110459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45661 (local address 127.0.97.3:37821)
0924 11:45:19.111242 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:19.111261 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:19.111383 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:19.111733 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:19.111738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:19.111763 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:19.112111 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.112122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.113196 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.113201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.114552 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:19.114563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.116303 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.116309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:19.116498 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.116530 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:45:19.116643 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 11:45:19.116747 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":1,"thread_start_us":353,"threads_started":1}
I20260924 11:45:19.117115 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.110445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37821 (local address 127.0.97.1:45661)
0924 11:45:19.110632 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.110639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.111344 (+ 705us) server_negotiation.cc:407] Connection header received
0924 11:45:19.111504 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.111509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.111594 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.111691 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:19.112334 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.113054 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.114733 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.115968 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.116093 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.116201 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.116702 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.116838 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:45:19.117286 18318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6baf16786c44bc846ed200bacb7bdb" candidate_uuid: "13a0688cd4724fc8bc0290fb6496aed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a901f43d8974b8ca008226529faa87f" is_pre_election: true
I20260924 11:45:19.117733 18318 raft_consensus.cc:2454] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a0688cd4724fc8bc0290fb6496aed6 in term 0.
I20260924 11:45:19.118407 18383 leader_election.cc:302] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a0688cd4724fc8bc0290fb6496aed6, 2a901f43d8974b8ca008226529faa87f; no voters:
I20260924 11:45:19.118289 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6baf16786c44bc846ed200bacb7bdb" candidate_uuid: "13a0688cd4724fc8bc0290fb6496aed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45391230093649e7823830b767c2e7d0" is_pre_election: true
I20260924 11:45:19.118799 18518 raft_consensus.cc:2781] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:45:19.118835 18188 raft_consensus.cc:2454] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a0688cd4724fc8bc0290fb6496aed6 in term 0.
I20260924 11:45:19.118940 18518 raft_consensus.cc:482] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:45:19.119038 18518 raft_consensus.cc:3037] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 0 FOLLOWER]: Advancing to term 1
W20260924 11:45:19.119132 18234 tablet.cc:2369] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:19.122246 18518 raft_consensus.cc:504] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.122835 18518 leader_election.cc:288] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [CANDIDATE]: Term 1 election: Requested vote from peers 45391230093649e7823830b767c2e7d0 (127.0.97.1:45661), 2a901f43d8974b8ca008226529faa87f (127.0.97.2:33389)
I20260924 11:45:19.123310 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6baf16786c44bc846ed200bacb7bdb" candidate_uuid: "13a0688cd4724fc8bc0290fb6496aed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45391230093649e7823830b767c2e7d0"
I20260924 11:45:19.123363 18318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6baf16786c44bc846ed200bacb7bdb" candidate_uuid: "13a0688cd4724fc8bc0290fb6496aed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a901f43d8974b8ca008226529faa87f"
I20260924 11:45:19.123507 18188 raft_consensus.cc:3037] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:19.123557 18318 raft_consensus.cc:3037] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:19.126021 18188 raft_consensus.cc:2454] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a0688cd4724fc8bc0290fb6496aed6 in term 1.
I20260924 11:45:19.126420 18382 leader_election.cc:302] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a0688cd4724fc8bc0290fb6496aed6, 45391230093649e7823830b767c2e7d0; no voters:
I20260924 11:45:19.126636 18318 raft_consensus.cc:2454] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a0688cd4724fc8bc0290fb6496aed6 in term 1.
I20260924 11:45:19.126698 18518 raft_consensus.cc:2781] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:19.127426 18518 raft_consensus.cc:686] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 1 LEADER]: Becoming Leader. State: Replica: 13a0688cd4724fc8bc0290fb6496aed6, State: Running, Role: LEADER
I20260924 11:45:19.128225 18518 consensus_queue.cc:237] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } }
I20260924 11:45:19.132618 18046 catalog_manager.cc:5495] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a0688cd4724fc8bc0290fb6496aed6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "13a0688cd4724fc8bc0290fb6496aed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13a0688cd4724fc8bc0290fb6496aed6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 } health_report { overall_health: UNKNOWN } } }
I20260924 11:45:19.157039 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.152598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.97.3:36963)
0924 11:45:19.152753 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.152757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.152772 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:45:19.152841 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.152844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.152913 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.153063 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:45:19.153340 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.153986 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.155203 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.155558 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.155638 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.156019 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:45:19.156521 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:45:19.156641 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.156713 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.156767 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:19.164582 18318 raft_consensus.cc:1261] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 1 FOLLOWER]: Refusing update from remote peer 13a0688cd4724fc8bc0290fb6496aed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:19.164582 18188 raft_consensus.cc:1261] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 1 FOLLOWER]: Refusing update from remote peer 13a0688cd4724fc8bc0290fb6496aed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:19.165526 18521 consensus_queue.cc:1035] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a901f43d8974b8ca008226529faa87f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:19.165869 18518 consensus_queue.cc:1035] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45391230093649e7823830b767c2e7d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:19.179370 18524 mvcc.cc:205] Tried to move back new op lower bound from 7332865307288199168 to 7332865307153338368. Current Snapshot: MvccSnapshot[applied={T|T < 7332865307288199168}]
I20260924 11:45:19.182493 18525 mvcc.cc:205] Tried to move back new op lower bound from 7332865307288199168 to 7332865307153338368. Current Snapshot: MvccSnapshot[applied={T|T < 7332865307288199168}]
I20260924 11:45:19.192018 18047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:19.192310 18047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:19.195865 18047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6cdf922ead449c28c88bafce557c461: Sending DeleteTablet for 3 replicas of tablet ad6baf16786c44bc846ed200bacb7bdb
I20260924 11:45:19.196949 18168 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6baf16786c44bc846ed200bacb7bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:19 UTC) from {username='slave'} at 127.0.0.1:60886
I20260924 11:45:19.196954 18428 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6baf16786c44bc846ed200bacb7bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:19 UTC) from {username='slave'} at 127.0.0.1:43386
I20260924 11:45:19.197674 18534 tablet_replica.cc:330] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: stopping tablet replica
I20260924 11:45:19.198096 18534 raft_consensus.cc:2229] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:19.198586 18534 raft_consensus.cc:2258] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:19.200539 18534 ts_tablet_manager.cc:1905] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:19.203009 18298 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6baf16786c44bc846ed200bacb7bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:19 UTC) from {username='slave'} at 127.0.0.1:34114
I20260924 11:45:19.203611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18106
I20260924 11:45:19.208489 18534 ts_tablet_manager.cc:1918] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:19.208660 18534 log.cc:1196] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/ad6baf16786c44bc846ed200bacb7bdb
I20260924 11:45:19.209218 18534 ts_tablet_manager.cc:1939] T ad6baf16786c44bc846ed200bacb7bdb P 45391230093649e7823830b767c2e7d0: Deleting consensus metadata
I20260924 11:45:19.211135 18032 catalog_manager.cc:4841] TS 45391230093649e7823830b767c2e7d0 (127.0.97.1:45661): tablet ad6baf16786c44bc846ed200bacb7bdb (table test-table [id=a15d542a601e4f4ab7a2ec354fbdce41]) successfully deleted
I20260924 11:45:19.221299 18536 tablet_replica.cc:330] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: stopping tablet replica
I20260924 11:45:19.221805 18536 raft_consensus.cc:2229] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:19.222374 18536 raft_consensus.cc:2258] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:19.224470 18536 ts_tablet_manager.cc:1905] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:19.228554 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.205478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.0.97.1:45661)
0924 11:45:19.205656 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.205662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.220647 (+ 14985us) server_negotiation.cc:407] Connection header received
0924 11:45:19.220873 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.220878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.220983 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.221091 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:19.221745 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.222834 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.224856 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.225474 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.225571 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.227000 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:19.227009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:19.227023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:19.227687 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:19.227743 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:19.227767 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:19.227852 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.228077 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.228220 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:45:19.229765 18228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:19.230048 18535 tablet_replica.cc:330] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: stopping tablet replica
I20260924 11:45:19.230742 18535 raft_consensus.cc:2229] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:45:19.231592 18535 raft_consensus.cc:2258] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:19.233817 18535 ts_tablet_manager.cc:1905] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:19.234076 18536 ts_tablet_manager.cc:1918] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:19.234233 18536 log.cc:1196] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/ad6baf16786c44bc846ed200bacb7bdb
I20260924 11:45:19.234722 18536 ts_tablet_manager.cc:1939] T ad6baf16786c44bc846ed200bacb7bdb P 2a901f43d8974b8ca008226529faa87f: Deleting consensus metadata
I20260924 11:45:19.236428 18033 catalog_manager.cc:4841] TS 2a901f43d8974b8ca008226529faa87f (127.0.97.2:33389): tablet ad6baf16786c44bc846ed200bacb7bdb (table test-table [id=a15d542a601e4f4ab7a2ec354fbdce41]) successfully deleted
I20260924 11:45:19.239737 18535 ts_tablet_manager.cc:1918] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:19.239898 18535 log.cc:1196] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/ad6baf16786c44bc846ed200bacb7bdb
I20260924 11:45:19.240408 18535 ts_tablet_manager.cc:1939] T ad6baf16786c44bc846ed200bacb7bdb P 13a0688cd4724fc8bc0290fb6496aed6: Deleting consensus metadata
I20260924 11:45:19.241760 18031 catalog_manager.cc:4841] TS 13a0688cd4724fc8bc0290fb6496aed6 (127.0.97.3:36963): tablet ad6baf16786c44bc846ed200bacb7bdb (table test-table [id=a15d542a601e4f4ab7a2ec354fbdce41]) successfully deleted
I20260924 11:45:19.369524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18106
I20260924 11:45:19.392867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18236
I20260924 11:45:19.399504 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.393700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.0.97.2:33389)
0924 11:45:19.393925 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.393934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.393955 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:45:19.394016 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.394027 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.394117 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.394222 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:19.394608 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.395312 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.396717 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.397096 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.397217 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.398241 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:19.398250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:19.398262 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:19.398811 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:19.398856 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:19.398873 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:19.398992 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.399115 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.399209 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:45:19.400272 18358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:19.483953 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18236
I20260924 11:45:19.500038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18366
I20260924 11:45:19.507335 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:19.500893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.97.3:36963)
0924 11:45:19.501104 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:19.501111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:19.501207 (+ 96us) server_negotiation.cc:407] Connection header received
0924 11:45:19.501427 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:19.501432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:19.501510 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:19.501617 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:19.502035 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.502916 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.504385 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:19.504825 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:19.504970 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:19.506066 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:19.506074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:19.506088 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:19.506613 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:19.506653 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:19.506672 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:19.506788 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:19.506915 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 11:45:19.507002 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:45:19.508121 18488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:19.625072 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18366
I20260924 11:45:19.644049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18016
I20260924 11:45:19.645191 18077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:19.747653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18016
2026-09-24T11:45:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2633 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:19 dist-test-slave-qhrh krb5kdc[18544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:19 dist-test-slave-qhrh krb5kdc[18544](info): set up 2 sockets
Sep 24 11:45:19 dist-test-slave-qhrh krb5kdc[18544](info): commencing operation
krb5kdc: starting...
W20260924 11:45:20.800092 388 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:20 dist-test-slave-qhrh krb5kdc[18544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:45:21.011533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37029
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:21.316524 18557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:21.316779 18557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:21.316967 18557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:21.323807 18557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:21.323902 18557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:21.323989 18557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:21.324105 18557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:21.324194 18557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:21.333225 18557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:21.334475 18557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:21.335721 18557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:21.346586 18562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:21.346536 18563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:21.346627 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:21.347337 18557 server_base.cc:1034] running on GCE node
I20260924 11:45:21.348054 18557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:21.349313 18557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:21.350512 18557 hybrid_clock.cc:648] HybridClock initialized: now 1790250321350460 us; error 36 us; skew 500 ppm
I20260924 11:45:21.350832 18557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:21 dist-test-slave-qhrh krb5kdc[18544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:21.353837 18557 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:45:21.355336 18557 webserver.cc:460] Webserver started at http://127.0.97.62:35549/ using document root <none> and password file <none>
I20260924 11:45:21.355976 18557 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:21.356120 18557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:21.356434 18557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:21.359216 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "45aba5eaaeec47648ea7cc7982135a44"
format_stamp: "Formatted at 2026-09-24 11:45:21 on dist-test-slave-qhrh"
server_key: "972d6344b182c5a88ce7ad06f498dadf"
server_key_iv: "1f78d14b1361c9a84e92da160b56b251"
server_key_version: "encryptionkey@0"
I20260924 11:45:21.359933 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "45aba5eaaeec47648ea7cc7982135a44"
format_stamp: "Formatted at 2026-09-24 11:45:21 on dist-test-slave-qhrh"
server_key: "972d6344b182c5a88ce7ad06f498dadf"
server_key_iv: "1f78d14b1361c9a84e92da160b56b251"
server_key_version: "encryptionkey@0"
I20260924 11:45:21.364890 18557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 11:45:21.368034 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:21.369308 18557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:45:21.369494 18557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "45aba5eaaeec47648ea7cc7982135a44"
format_stamp: "Formatted at 2026-09-24 11:45:21 on dist-test-slave-qhrh"
server_key: "972d6344b182c5a88ce7ad06f498dadf"
server_key_iv: "1f78d14b1361c9a84e92da160b56b251"
server_key_version: "encryptionkey@0"
I20260924 11:45:21.369678 18557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:21.382155 18557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:21.385679 18557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:21.385960 18557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:21.404208 18557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37029
I20260924 11:45:21.404240 18623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37029 every 8 connection(s)
I20260924 11:45:21.405491 18557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:21.408957 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:21.414951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18557
I20260924 11:45:21.415138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:21.415660 388 external_mini_cluster.cc:1428] Setting key bd07496e9ba8ef82a6cd872cdeb2f0f5
I20260924 11:45:21.420032 18624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44: Bootstrap starting.
I20260924 11:45:21.423619 18624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:21.424875 18624 log.cc:824] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:21.427820 18624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44: No bootstrap required, opened a new log
Sep 24 11:45:21 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:21.435485 18624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aba5eaaeec47648ea7cc7982135a44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37029 } }
I20260924 11:45:21.435853 18624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:21.435981 18624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aba5eaaeec47648ea7cc7982135a44, State: Initialized, Role: FOLLOWER
I20260924 11:45:21.436589 18624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aba5eaaeec47648ea7cc7982135a44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37029 } }
I20260924 11:45:21.436836 18624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:21.436991 18624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:21.437183 18624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:21.438054 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:21.417513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.0.97.62:37029)
0924 11:45:21.418379 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:21.418389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:21.418427 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:45:21.420025 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:21.420044 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:21.420447 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:21.420768 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:21.422543 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:21.423868 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:21.425760 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:21.426484 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:21.426615 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:21.431160 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:21.431168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:21.431181 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:21.431211 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:21.434247 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:21.434890 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:21.434897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:21.434910 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:21.435052 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:21.435390 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:21.435395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:21.435397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:21.435945 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:21.436249 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:21.436662 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 11:45:21.437065 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":437,"threads_started":1}
I20260924 11:45:21.439299 18624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aba5eaaeec47648ea7cc7982135a44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37029 } }
I20260924 11:45:21.439750 18624 leader_election.cc:302] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45aba5eaaeec47648ea7cc7982135a44; no voters:
I20260924 11:45:21.440419 18624 leader_election.cc:288] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:21.440541 18629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:21.441278 18629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [term 1 LEADER]: Becoming Leader. State: Replica: 45aba5eaaeec47648ea7cc7982135a44, State: Running, Role: LEADER
I20260924 11:45:21.441852 18629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aba5eaaeec47648ea7cc7982135a44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37029 } }
I20260924 11:45:21.442602 18624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:21.445554 18630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45aba5eaaeec47648ea7cc7982135a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aba5eaaeec47648ea7cc7982135a44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37029 } } }
I20260924 11:45:21.445636 18631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45aba5eaaeec47648ea7cc7982135a44. Latest consensus state: current_term: 1 leader_uuid: "45aba5eaaeec47648ea7cc7982135a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aba5eaaeec47648ea7cc7982135a44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37029 } } }
I20260924 11:45:21.445961 18631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:21.445864 18630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:21.447685 18635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:21.453421 18635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:21.462100 18635 catalog_manager.cc:1347] Generated new cluster ID: 82037fc26b9e44999b3ed694d5d4e398
I20260924 11:45:21.462203 18635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:21.480015 18635 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:21.481345 18635 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:21.490849 18635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44: Generated new TSK 0
I20260924 11:45:21.491636 18635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:45:21.545408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37029
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:21.849385 18650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:21.849622 18650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:21.849705 18650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:21.856623 18650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:21.856714 18650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:21.856814 18650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:45:21.865489 18650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:21.866537 18650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:21.867727 18650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:21.880422 18655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:21.880419 18656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:21.880864 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:21.881222 18650 server_base.cc:1034] running on GCE node
I20260924 11:45:21.881806 18650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:21.882702 18650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:21.883914 18650 hybrid_clock.cc:648] HybridClock initialized: now 1790250321883862 us; error 46 us; skew 500 ppm
I20260924 11:45:21.884274 18650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:21 dist-test-slave-qhrh krb5kdc[18544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:21.887593 18650 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:45:21.889164 18650 webserver.cc:460] Webserver started at http://127.0.97.1:41493/ using document root <none> and password file <none>
I20260924 11:45:21.889806 18650 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:21.889936 18650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:21.890225 18650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:21.893103 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "fb07b35a2f3243059400ad0042044755"
format_stamp: "Formatted at 2026-09-24 11:45:21 on dist-test-slave-qhrh"
server_key: "77387242cb478030798c811f96b4e4f3"
server_key_iv: "c287b7770c0744bb8f1bb19b96a1d68c"
server_key_version: "encryptionkey@0"
I20260924 11:45:21.893821 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fb07b35a2f3243059400ad0042044755"
format_stamp: "Formatted at 2026-09-24 11:45:21 on dist-test-slave-qhrh"
server_key: "77387242cb478030798c811f96b4e4f3"
server_key_iv: "c287b7770c0744bb8f1bb19b96a1d68c"
server_key_version: "encryptionkey@0"
I20260924 11:45:21.898227 18650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 11:45:21.901312 18665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:21.902529 18650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:21.902714 18650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "fb07b35a2f3243059400ad0042044755"
format_stamp: "Formatted at 2026-09-24 11:45:21 on dist-test-slave-qhrh"
server_key: "77387242cb478030798c811f96b4e4f3"
server_key_iv: "c287b7770c0744bb8f1bb19b96a1d68c"
server_key_version: "encryptionkey@0"
I20260924 11:45:21.902902 18650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:21.933398 18650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:21.936966 18650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:21.937239 18650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:21.938087 18650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:21.939230 18650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:21.939320 18650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:21.939455 18650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:21.939515 18650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:21.970469 18650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41423
I20260924 11:45:21.970553 18777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41423 every 8 connection(s)
I20260924 11:45:21.971784 18650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:45:21.975703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18650
I20260924 11:45:21.975869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:45:21.976341 388 external_mini_cluster.cc:1428] Setting key 5d125868e16daa1a53a6ab35bc9eced9
Sep 24 11:45:21 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:21.991253 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:21.974243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45353 (local address 127.0.97.62:37029)
0924 11:45:21.974711 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:21.974716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:21.975711 (+ 995us) server_negotiation.cc:407] Connection header received
0924 11:45:21.977551 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:21.977572 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:21.977671 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:21.977768 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:21.979215 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:21.980277 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:21.981766 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:21.982485 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:21.982637 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:21.985658 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:21.985666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:21.985671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:21.985711 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:21.988109 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:21.988971 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:21.988974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:21.988976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:21.989093 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:21.989515 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:21.989517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:21.989519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:21.989767 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:21.989864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:21.990730 (+ 866us) server_negotiation.cc:299] Negotiation successful
0924 11:45:21.990884 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":186,"threads_started":1}
I20260924 11:45:21.992481 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:21.974935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37029 (local address 127.0.97.1:45353)
0924 11:45:21.975572 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:21.975611 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:21.977334 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:21.978033 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:21.978051 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:21.978509 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:21.979045 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:21.979063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:21.980398 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:21.980401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:21.981625 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:21.981637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:21.982694 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:21.982698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:21.982866 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:21.983479 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:21.983499 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:21.985458 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:21.988290 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:21.988301 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:21.988317 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:21.988877 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:21.989196 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:21.989203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:21.989208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:21.989425 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:21.989878 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:21.989887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:21.990288 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 11:45:21.990591 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 11:45:21.991098 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
I20260924 11:45:21.993911 18778 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37029
I20260924 11:45:21.994189 18778 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:21.994848 18778 heartbeater.cc:511] Master 127.0.97.62:37029 requested a full tablet report, sending...
I20260924 11:45:21.996950 18589 ts_manager.cc:194] Registered new tserver with Master: fb07b35a2f3243059400ad0042044755 (127.0.97.1:41423)
I20260924 11:45:21.998252 18589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45353
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:45:22.028345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37029
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:22.341063 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:22.341352 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:22.341535 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:22.348366 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:22.348476 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:22.348618 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:45:22.357275 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:22.358450 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:22.359663 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:22.371709 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:22.371804 18792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:22.371749 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:22.372586 18784 server_base.cc:1034] running on GCE node
I20260924 11:45:22.373229 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:22.374130 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:22.375361 18784 hybrid_clock.cc:648] HybridClock initialized: now 1790250322375312 us; error 42 us; skew 500 ppm
I20260924 11:45:22.375667 18784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:22 dist-test-slave-qhrh krb5kdc[18544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:22.379832 18784 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:45:22.381453 18784 webserver.cc:460] Webserver started at http://127.0.97.2:43779/ using document root <none> and password file <none>
I20260924 11:45:22.382102 18784 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:22.382232 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:22.382537 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:22.385102 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "4a3f40fac7ba4b5e903ae7e5be22608f"
format_stamp: "Formatted at 2026-09-24 11:45:22 on dist-test-slave-qhrh"
server_key: "b84a55ae70a709fa183c6f17b1dc189d"
server_key_iv: "a8d08a5c3c418957b3cc7eefb6941e23"
server_key_version: "encryptionkey@0"
I20260924 11:45:22.385834 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4a3f40fac7ba4b5e903ae7e5be22608f"
format_stamp: "Formatted at 2026-09-24 11:45:22 on dist-test-slave-qhrh"
server_key: "b84a55ae70a709fa183c6f17b1dc189d"
server_key_iv: "a8d08a5c3c418957b3cc7eefb6941e23"
server_key_version: "encryptionkey@0"
I20260924 11:45:22.390661 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:22.393798 18799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:22.394919 18784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:22.395081 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "4a3f40fac7ba4b5e903ae7e5be22608f"
format_stamp: "Formatted at 2026-09-24 11:45:22 on dist-test-slave-qhrh"
server_key: "b84a55ae70a709fa183c6f17b1dc189d"
server_key_iv: "a8d08a5c3c418957b3cc7eefb6941e23"
server_key_version: "encryptionkey@0"
I20260924 11:45:22.395258 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:22.417107 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:22.420624 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:22.420905 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:22.421809 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:22.423015 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:22.423118 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:22.423251 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:22.423316 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:22.453086 18784 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45473
I20260924 11:45:22.453164 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45473 every 8 connection(s)
I20260924 11:45:22.454306 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:45:22.457569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18784
I20260924 11:45:22.457782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:45:22.458313 388 external_mini_cluster.cc:1428] Setting key 92607f845a8d23d03216453d9bf632b7
Sep 24 11:45:22 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:22.474411 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:22.456786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53027 (local address 127.0.97.62:37029)
0924 11:45:22.456961 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:22.456966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:22.458278 (+ 1312us) server_negotiation.cc:407] Connection header received
0924 11:45:22.460412 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:22.460417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:22.460492 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:22.460614 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:22.461990 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.463165 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.464686 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.465406 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.465560 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:22.468869 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:22.468874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:22.468877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:22.468906 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:22.471383 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:22.472094 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:22.472098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:22.472101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:22.472284 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:22.472715 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:22.472719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:22.472734 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:22.472982 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:22.473130 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:22.473930 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 11:45:22.474131 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:45:22.475808 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:22.457297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37029 (local address 127.0.97.2:53027)
0924 11:45:22.458133 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:22.458183 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:22.460047 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:22.460797 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:22.460805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:22.461294 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:22.461818 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:22.461835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.463297 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.463300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:22.464557 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:22.464567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.465606 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.465611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:22.465781 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:22.466423 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:22.466443 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:22.468667 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:22.471531 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:22.471539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:22.471553 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:22.471955 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:22.472392 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:22.472398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:22.472401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:22.472587 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:22.473129 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:22.473138 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:22.473538 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 11:45:22.473837 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 11:45:22.474395 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":345,"threads_started":1}
I20260924 11:45:22.477286 18912 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37029
I20260924 11:45:22.477573 18912 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:22.478374 18912 heartbeater.cc:511] Master 127.0.97.62:37029 requested a full tablet report, sending...
I20260924 11:45:22.479902 18589 ts_manager.cc:194] Registered new tserver with Master: 4a3f40fac7ba4b5e903ae7e5be22608f (127.0.97.2:45473)
I20260924 11:45:22.480728 18589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53027
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:45:22.510918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37029
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:22.811924 18917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:22.812242 18917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:22.812346 18917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:22.819082 18917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:22.819178 18917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:22.819357 18917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:45:22.828300 18917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:22.829458 18917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:22.830688 18917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:22.843271 18922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:22.843223 18923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:22.843299 18925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:22.844347 18917 server_base.cc:1034] running on GCE node
I20260924 11:45:22.844906 18917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:22.845726 18917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:22.846908 18917 hybrid_clock.cc:648] HybridClock initialized: now 1790250322846862 us; error 35 us; skew 500 ppm
I20260924 11:45:22.847213 18917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:22 dist-test-slave-qhrh krb5kdc[18544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:22.859395 18917 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:45:22.860893 18917 webserver.cc:460] Webserver started at http://127.0.97.3:33775/ using document root <none> and password file <none>
I20260924 11:45:22.861564 18917 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:22.861711 18917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:22.862016 18917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:22.864840 18917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "0f6bb2dd10ab4cab83a2899b5f13302e"
format_stamp: "Formatted at 2026-09-24 11:45:22 on dist-test-slave-qhrh"
server_key: "f5c77ea8dea4339db2fefbcad28cf1af"
server_key_iv: "715044b990c88224732967d06a8f8cf0"
server_key_version: "encryptionkey@0"
I20260924 11:45:22.865571 18917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0f6bb2dd10ab4cab83a2899b5f13302e"
format_stamp: "Formatted at 2026-09-24 11:45:22 on dist-test-slave-qhrh"
server_key: "f5c77ea8dea4339db2fefbcad28cf1af"
server_key_iv: "715044b990c88224732967d06a8f8cf0"
server_key_version: "encryptionkey@0"
I20260924 11:45:22.870191 18917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:22.873445 18932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:22.874687 18917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:45:22.874837 18917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "0f6bb2dd10ab4cab83a2899b5f13302e"
format_stamp: "Formatted at 2026-09-24 11:45:22 on dist-test-slave-qhrh"
server_key: "f5c77ea8dea4339db2fefbcad28cf1af"
server_key_iv: "715044b990c88224732967d06a8f8cf0"
server_key_version: "encryptionkey@0"
I20260924 11:45:22.875002 18917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:22.896895 18917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:22.900852 18917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:22.901124 18917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:22.902027 18917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:22.903255 18917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:22.903322 18917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:22.903474 18917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:22.903519 18917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:22.936062 18917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40817
I20260924 11:45:22.936187 19044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40817 every 8 connection(s)
I20260924 11:45:22.937539 18917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:45:22.941354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 18917
I20260924 11:45:22.941540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:45:22.941987 388 external_mini_cluster.cc:1428] Setting key dfed5482f48e19b798d4d1e0f8a6db85
Sep 24 11:45:22 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:22.955626 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:22.939800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55453 (local address 127.0.97.62:37029)
0924 11:45:22.939964 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:22.939968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:22.941036 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 11:45:22.942932 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:22.942937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:22.943022 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:22.943158 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:22.944709 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.945435 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.946825 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.947207 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.947327 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:22.950344 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:22.950349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:22.950351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:22.950379 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:22.952809 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:22.953456 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:22.953462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:22.953467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:22.953607 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:22.953999 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:22.954004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:22.954007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:22.954274 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:22.954419 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:22.955055 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 11:45:22.955296 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:45:22.956658 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:22.940225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37029 (local address 127.0.97.3:55453)
0924 11:45:22.940880 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:22.940920 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:22.942664 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:22.943361 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:22.943368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:22.943827 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:22.944554 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:22.944572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.945585 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.945589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:22.946687 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:22.946697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.947404 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.947411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:22.947623 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:22.948287 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:22.948308 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:22.950159 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:22.952950 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:22.952958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:22.952972 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:22.953343 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:22.953712 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:22.953717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:22.953721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:22.953885 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:22.954405 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:22.954412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:22.954713 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 11:45:22.954934 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 11:45:22.955426 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":229,"threads_started":1}
I20260924 11:45:22.957864 19045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37029
I20260924 11:45:22.958141 19045 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:22.958786 19045 heartbeater.cc:511] Master 127.0.97.62:37029 requested a full tablet report, sending...
I20260924 11:45:22.959990 18589 ts_manager.cc:194] Registered new tserver with Master: 0f6bb2dd10ab4cab83a2899b5f13302e (127.0.97.3:40817)
I20260924 11:45:22.960805 18589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55453
I20260924 11:45:22.968838 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:45:22.985322 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:22.972994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.0.97.62:37029)
0924 11:45:22.973169 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:22.973174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:22.973697 (+ 523us) server_negotiation.cc:407] Connection header received
0924 11:45:22.973857 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:22.973861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:22.973914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:22.973990 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:22.975777 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.976562 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.977877 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:22.978241 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:22.978322 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:22.980751 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:22.980755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:22.980758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:22.980795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:22.983093 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:22.983672 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:22.983675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:22.983677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:22.983774 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:22.984094 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:22.984097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:22.984099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:22.984392 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:22.984577 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:22.984838 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 11:45:22.984984 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:45:22.988659 18589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:45:22.994839 18589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:45:23.001091 18778 heartbeater.cc:503] Master 127.0.97.62:37029 was elected leader, sending a full tablet report...
I20260924 11:45:23.036286 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.007062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.0.97.2:45473)
0924 11:45:23.026493 (+ 19431us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.026507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.028049 (+ 1542us) server_negotiation.cc:407] Connection header received
0924 11:45:23.028201 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.028209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.028497 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.028695 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:23.029821 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.031311 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.033584 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.035065 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.035196 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.035409 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.035788 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.035945 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19242,"thread_start_us":19129,"threads_started":1}
I20260924 11:45:23.036286 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.006689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45473 (local address 127.0.0.1:52496)
0924 11:45:23.027073 (+ 20384us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:23.027132 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:23.027315 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:23.029349 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:23.029354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:23.029384 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:23.029664 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.029673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.031462 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.031466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.033382 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.033414 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.035273 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.035281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.035533 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.035609 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 11:45:23.035753 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:45:23.035931 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20148,"thread_start_us":20805,"threads_started":1}
I20260924 11:45:23.038095 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.006689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41423 (local address 127.0.0.1:48144)
0924 11:45:23.030232 (+ 23543us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:23.030250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:23.030364 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:23.031194 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:23.031198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:23.031223 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:23.031743 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.031759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.033564 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.033569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.035500 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.035514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.037380 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.037386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.037553 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.037584 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:45:23.037679 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:45:23.037781 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23399,"mutex_wait_us":566,"thread_start_us":19491,"threads_started":1}
I20260924 11:45:23.038195 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.006813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.0.97.1:41423)
0924 11:45:23.029265 (+ 22452us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.029272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.030363 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 11:45:23.030491 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.030498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.030749 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.030943 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:23.031961 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.033432 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.035670 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.037174 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.037307 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.037524 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.037732 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.037899 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22314,"thread_start_us":22166,"threads_started":1}
I20260924 11:45:23.038522 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.007052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40817 (local address 127.0.0.1:52354)
0924 11:45:23.027774 (+ 20722us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:23.027790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:23.027901 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:23.030459 (+ 2558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:23.030464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:23.030489 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:23.030872 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.030889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.033133 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.033138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.035793 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.035807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.037551 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.037557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.037700 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.037727 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 11:45:23.037786 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 11:45:23.037908 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20587,"mutex_wait_us":380,"thread_start_us":22898,"threads_started":1}
I20260924 11:45:23.039422 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.007776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.0.97.3:40817)
0924 11:45:23.029792 (+ 22016us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.029804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.029844 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:45:23.029955 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.029962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.030222 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.031058 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:23.031365 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.032976 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.035964 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.037338 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.038756 (+ 1418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.038926 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.038974 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.039077 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21823,"thread_start_us":21676,"threads_started":1}
I20260924 11:45:23.042311 18713 tablet_service.cc:1467] Processing CreateTablet for tablet 3aceae213ae644d58fdf91930d376478 (DEFAULT_TABLE table=test-table [id=1915f524b10e4bb599ff8f4618cd8a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:23.042841 18847 tablet_service.cc:1467] Processing CreateTablet for tablet 3aceae213ae644d58fdf91930d376478 (DEFAULT_TABLE table=test-table [id=1915f524b10e4bb599ff8f4618cd8a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:23.042788 18980 tablet_service.cc:1467] Processing CreateTablet for tablet 3aceae213ae644d58fdf91930d376478 (DEFAULT_TABLE table=test-table [id=1915f524b10e4bb599ff8f4618cd8a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:23.043843 18713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aceae213ae644d58fdf91930d376478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:23.043996 18980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aceae213ae644d58fdf91930d376478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:23.044046 18847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aceae213ae644d58fdf91930d376478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:23.054499 19063 tablet_bootstrap.cc:492] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Bootstrap starting.
I20260924 11:45:23.057997 19065 tablet_bootstrap.cc:492] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Bootstrap starting.
I20260924 11:45:23.058068 19063 tablet_bootstrap.cc:654] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:23.059360 19063 log.cc:824] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:23.059502 19064 tablet_bootstrap.cc:492] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Bootstrap starting.
I20260924 11:45:23.061344 19065 tablet_bootstrap.cc:654] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:23.062477 19064 tablet_bootstrap.cc:654] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:23.062547 19065 log.cc:824] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:23.062870 19063 tablet_bootstrap.cc:492] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: No bootstrap required, opened a new log
I20260924 11:45:23.063151 19063 ts_tablet_manager.cc:1397] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260924 11:45:23.063969 19064 log.cc:824] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:23.066583 19065 tablet_bootstrap.cc:492] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: No bootstrap required, opened a new log
I20260924 11:45:23.066782 19064 tablet_bootstrap.cc:492] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: No bootstrap required, opened a new log
I20260924 11:45:23.066855 19065 ts_tablet_manager.cc:1397] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 11:45:23.067006 19064 ts_tablet_manager.cc:1397] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 11:45:23.073501 19063 raft_consensus.cc:348] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.074075 19063 raft_consensus.cc:374] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:23.074180 19063 raft_consensus.cc:729] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3f40fac7ba4b5e903ae7e5be22608f, State: Initialized, Role: FOLLOWER
I20260924 11:45:23.074777 19065 raft_consensus.cc:348] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.074813 19063 consensus_queue.cc:260] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.075129 19065 raft_consensus.cc:374] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:23.075243 19065 raft_consensus.cc:729] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6bb2dd10ab4cab83a2899b5f13302e, State: Initialized, Role: FOLLOWER
I20260924 11:45:23.075922 18912 heartbeater.cc:503] Master 127.0.97.62:37029 was elected leader, sending a full tablet report...
I20260924 11:45:23.075855 19065 consensus_queue.cc:260] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.076728 19063 ts_tablet_manager.cc:1428] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 11:45:23.077512 19045 heartbeater.cc:503] Master 127.0.97.62:37029 was elected leader, sending a full tablet report...
I20260924 11:45:23.077555 19064 raft_consensus.cc:348] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.077929 19064 raft_consensus.cc:374] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:23.078009 19064 raft_consensus.cc:729] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb07b35a2f3243059400ad0042044755, State: Initialized, Role: FOLLOWER
I20260924 11:45:23.078178 19065 ts_tablet_manager.cc:1428] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 11:45:23.078569 19064 consensus_queue.cc:260] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.080448 19064 ts_tablet_manager.cc:1428] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 11:45:23.114910 19071 raft_consensus.cc:482] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:23.115172 19071 raft_consensus.cc:504] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.116777 19071 leader_election.cc:288] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a3f40fac7ba4b5e903ae7e5be22608f (127.0.97.2:45473), 0f6bb2dd10ab4cab83a2899b5f13302e (127.0.97.3:40817)
I20260924 11:45:23.123308 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.117166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38019 (local address 127.0.97.2:45473)
0924 11:45:23.117372 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.117379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.118113 (+ 734us) server_negotiation.cc:407] Connection header received
0924 11:45:23.118337 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.118340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.118412 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.118504 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:23.119127 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.119826 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.121627 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.122368 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.122523 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.122632 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.122923 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.123038 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:45:23.123345 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.117335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38083 (local address 127.0.97.3:40817)
0924 11:45:23.117555 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.117563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.118197 (+ 634us) server_negotiation.cc:407] Connection header received
0924 11:45:23.118408 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.118412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.118465 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.118553 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:23.119186 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.119911 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.121677 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.122423 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.122572 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.122679 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.122954 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.123069 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:45:23.123494 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.117187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40817 (local address 127.0.97.1:38083)
0924 11:45:23.118091 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:23.118105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:23.118246 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:23.118578 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:23.118582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:23.118608 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:23.118997 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.119007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.120048 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.120053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.121519 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.121529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.122650 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.122656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.122808 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.122829 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:45:23.122930 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:45:23.123043 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":16,"thread_start_us":479,"threads_started":1}
I20260924 11:45:23.123719 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.117138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45473 (local address 127.0.97.1:38019)
0924 11:45:23.118004 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:23.118019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:23.118185 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:23.118518 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:23.118522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:23.118544 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:23.118980 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.118992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.119949 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.119956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.121452 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:23.121463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.122596 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.122602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:23.122751 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.122788 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:45:23.122889 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:45:23.123003 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":1,"spinlock_wait_cycles":15232,"thread_start_us":507,"threads_started":1}
I20260924 11:45:23.124533 19000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aceae213ae644d58fdf91930d376478" candidate_uuid: "fb07b35a2f3243059400ad0042044755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" is_pre_election: true
I20260924 11:45:23.124665 18867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aceae213ae644d58fdf91930d376478" candidate_uuid: "fb07b35a2f3243059400ad0042044755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" is_pre_election: true
I20260924 11:45:23.125020 19000 raft_consensus.cc:2454] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb07b35a2f3243059400ad0042044755 in term 0.
I20260924 11:45:23.125144 18867 raft_consensus.cc:2454] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb07b35a2f3243059400ad0042044755 in term 0.
I20260924 11:45:23.125669 18666 leader_election.cc:302] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f6bb2dd10ab4cab83a2899b5f13302e, 4a3f40fac7ba4b5e903ae7e5be22608f, fb07b35a2f3243059400ad0042044755; no voters:
I20260924 11:45:23.126048 19071 raft_consensus.cc:2781] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:45:23.126191 19071 raft_consensus.cc:482] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:45:23.126307 19071 raft_consensus.cc:3037] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:23.128495 19071 raft_consensus.cc:504] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.129033 19071 leader_election.cc:288] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [CANDIDATE]: Term 1 election: Requested vote from peers 4a3f40fac7ba4b5e903ae7e5be22608f (127.0.97.2:45473), 0f6bb2dd10ab4cab83a2899b5f13302e (127.0.97.3:40817)
I20260924 11:45:23.129462 18867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aceae213ae644d58fdf91930d376478" candidate_uuid: "fb07b35a2f3243059400ad0042044755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f"
I20260924 11:45:23.129522 19000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aceae213ae644d58fdf91930d376478" candidate_uuid: "fb07b35a2f3243059400ad0042044755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e"
I20260924 11:45:23.129663 18867 raft_consensus.cc:3037] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:23.129737 19000 raft_consensus.cc:3037] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:23.132311 18867 raft_consensus.cc:2454] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb07b35a2f3243059400ad0042044755 in term 1.
I20260924 11:45:23.132452 19000 raft_consensus.cc:2454] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb07b35a2f3243059400ad0042044755 in term 1.
I20260924 11:45:23.132694 18667 leader_election.cc:302] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3f40fac7ba4b5e903ae7e5be22608f, fb07b35a2f3243059400ad0042044755; no voters:
I20260924 11:45:23.132985 19071 raft_consensus.cc:2781] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:23.133379 19071 raft_consensus.cc:686] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 1 LEADER]: Becoming Leader. State: Replica: fb07b35a2f3243059400ad0042044755, State: Running, Role: LEADER
I20260924 11:45:23.133863 19071 consensus_queue.cc:237] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } }
I20260924 11:45:23.138017 18588 catalog_manager.cc:5495] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb07b35a2f3243059400ad0042044755 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fb07b35a2f3243059400ad0042044755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb07b35a2f3243059400ad0042044755" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 } health_report { overall_health: UNKNOWN } } }
I20260924 11:45:23.166967 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.161606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.0.97.1:41423)
0924 11:45:23.161808 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.161814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.161906 (+ 92us) server_negotiation.cc:407] Connection header received
0924 11:45:23.162140 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.162144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.162217 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.162365 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:45:23.162753 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.163467 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.164890 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.165264 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.165387 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.165963 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:45:23.166405 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:45:23.166584 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.166691 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.166734 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:45:23.174820 18867 raft_consensus.cc:1261] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 1 FOLLOWER]: Refusing update from remote peer fb07b35a2f3243059400ad0042044755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:23.174822 19000 raft_consensus.cc:1261] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 1 FOLLOWER]: Refusing update from remote peer fb07b35a2f3243059400ad0042044755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:23.175671 19074 consensus_queue.cc:1035] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3f40fac7ba4b5e903ae7e5be22608f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:23.176002 19071 consensus_queue.cc:1035] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6bb2dd10ab4cab83a2899b5f13302e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:23.188423 19078 mvcc.cc:205] Tried to move back new op lower bound from 7332865323714555904 to 7332865323560415232. Current Snapshot: MvccSnapshot[applied={T|T < 7332865323714555904}]
I20260924 11:45:23.198695 18587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:23.199030 18587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:23.202554 18587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45aba5eaaeec47648ea7cc7982135a44: Sending DeleteTablet for 3 replicas of tablet 3aceae213ae644d58fdf91930d376478
I20260924 11:45:23.203691 18713 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aceae213ae644d58fdf91930d376478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48144
I20260924 11:45:23.203698 18847 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aceae213ae644d58fdf91930d376478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52496
I20260924 11:45:23.203835 18980 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aceae213ae644d58fdf91930d376478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52354
I20260924 11:45:23.204901 19087 tablet_replica.cc:330] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: stopping tablet replica
I20260924 11:45:23.205368 19087 raft_consensus.cc:2229] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:45:23.206058 19087 raft_consensus.cc:2258] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:23.206841 19088 tablet_replica.cc:330] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: stopping tablet replica
I20260924 11:45:23.208346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18650
I20260924 11:45:23.210009 19087 ts_tablet_manager.cc:1905] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:23.214294 19088 raft_consensus.cc:2229] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:23.214805 19088 raft_consensus.cc:2258] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:23.218092 19088 ts_tablet_manager.cc:1905] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:23.218861 19087 ts_tablet_manager.cc:1918] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:23.225063 19087 log.cc:1196] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/3aceae213ae644d58fdf91930d376478
I20260924 11:45:23.225754 19087 ts_tablet_manager.cc:1939] T 3aceae213ae644d58fdf91930d376478 P fb07b35a2f3243059400ad0042044755: Deleting consensus metadata
I20260924 11:45:23.228268 19088 ts_tablet_manager.cc:1918] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:23.228904 19089 tablet_replica.cc:330] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: stopping tablet replica
I20260924 11:45:23.229295 19089 raft_consensus.cc:2229] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:23.229729 19089 raft_consensus.cc:2258] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:23.230115 19088 log.cc:1196] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/3aceae213ae644d58fdf91930d376478
I20260924 11:45:23.230552 19088 ts_tablet_manager.cc:1939] T 3aceae213ae644d58fdf91930d376478 P 4a3f40fac7ba4b5e903ae7e5be22608f: Deleting consensus metadata
Sep 24 11:45:23 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:45:23.232038 18576 catalog_manager.cc:4841] TS fb07b35a2f3243059400ad0042044755 (127.0.97.1:41423): tablet 3aceae213ae644d58fdf91930d376478 (table test-table [id=1915f524b10e4bb599ff8f4618cd8a52]) successfully deleted
I20260924 11:45:23.232244 18574 catalog_manager.cc:4841] TS 4a3f40fac7ba4b5e903ae7e5be22608f (127.0.97.2:45473): tablet 3aceae213ae644d58fdf91930d376478 (table test-table [id=1915f524b10e4bb599ff8f4618cd8a52]) successfully deleted
I20260924 11:45:23.233292 19089 ts_tablet_manager.cc:1905] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:23.239806 19089 ts_tablet_manager.cc:1918] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:23.240000 19089 log.cc:1196] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/3aceae213ae644d58fdf91930d376478
I20260924 11:45:23.240572 19089 ts_tablet_manager.cc:1939] T 3aceae213ae644d58fdf91930d376478 P 0f6bb2dd10ab4cab83a2899b5f13302e: Deleting consensus metadata
I20260924 11:45:23.240691 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.210958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.0.97.1:41423)
0924 11:45:23.211175 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.211183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.219451 (+ 8268us) server_negotiation.cc:407] Connection header received
0924 11:45:23.219748 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.219756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.219853 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.219932 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:23.221710 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.222916 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.225951 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.226650 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.226741 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.233628 (+ 6887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:23.233642 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:23.233658 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:23.233698 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:23.237615 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:23.238410 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:23.238418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:23.238423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:23.238574 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:23.238987 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:23.238992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:23.238995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:23.239394 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:23.239678 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.239981 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.240304 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:45:23.241376 18773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:23.242367 18573 catalog_manager.cc:4841] TS 0f6bb2dd10ab4cab83a2899b5f13302e (127.0.97.3:40817): tablet 3aceae213ae644d58fdf91930d376478 (table test-table [id=1915f524b10e4bb599ff8f4618cd8a52]) successfully deleted
I20260924 11:45:23.335383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18650
I20260924 11:45:23.353000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18784
Sep 24 11:45:23 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:45:23.367031 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.353839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.97.2:45473)
0924 11:45:23.354011 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.354016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.354034 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:45:23.354164 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.354168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.354227 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.354310 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:23.355380 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.356268 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.357441 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.357832 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.357925 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.361938 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:23.361951 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:23.361966 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:23.361996 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:23.364910 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:23.365497 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:23.365503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:23.365506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:23.365608 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:23.365896 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:23.365899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:23.365901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:23.366216 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:23.366419 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.366612 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.366754 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:45:23.367686 18907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:23.459973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18784
I20260924 11:45:23.476526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18917
Sep 24 11:45:23 dist-test-slave-qhrh krb5kdc[18544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:45:23.490691 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:23.477193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.0.97.3:40817)
0924 11:45:23.477400 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:23.477406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:23.477424 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:45:23.477570 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:23.477576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:23.477650 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:23.477759 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:23.479000 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.479741 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.481037 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:23.481431 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:23.481560 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:23.485754 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:23.485766 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:23.485782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:23.485810 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:23.488066 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:23.488753 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:23.488769 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:23.488772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:23.488924 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:23.489328 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:23.489333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:23.489336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:23.489663 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:23.489943 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:23.490206 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 11:45:23.490406 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:45:23.491348 19040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:23.577272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18917
I20260924 11:45:23.592502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 18557
I20260924 11:45:23.593526 18619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:23.699265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 18557
2026-09-24T11:45:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3952 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T11:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:23Z Disabled control of system clock
I20260924 11:45:23.770397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34395
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:45:24.078115 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:24.078523 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:24.078681 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:24.085803 19098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:24.085935 19098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:24.086014 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:24.086112 19098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:24.086199 19098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:24.094455 19098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:24.095606 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:24.097358 19098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:24.108421 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:24.108479 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:24.108381 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:24.109356 19098 server_base.cc:1034] running on GCE node
I20260924 11:45:24.110088 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:24.111325 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:24.112532 19098 hybrid_clock.cc:648] HybridClock initialized: now 1790250324112500 us; error 43 us; skew 500 ppm
I20260924 11:45:24.112859 19098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:24.114918 19098 webserver.cc:460] Webserver started at http://127.0.97.62:34855/ using document root <none> and password file <none>
I20260924 11:45:24.115561 19098 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:24.115686 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:24.116037 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:24.118919 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "907f47e3d69d414bad230c3a9e7d49be"
format_stamp: "Formatted at 2026-09-24 11:45:24 on dist-test-slave-qhrh"
server_key: "c51196f6a872d8fed63246850767b09b"
server_key_iv: "d65f5f32d26b00ce4b3cfab14a648b2d"
server_key_version: "encryptionkey@0"
I20260924 11:45:24.119676 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "907f47e3d69d414bad230c3a9e7d49be"
format_stamp: "Formatted at 2026-09-24 11:45:24 on dist-test-slave-qhrh"
server_key: "c51196f6a872d8fed63246850767b09b"
server_key_iv: "d65f5f32d26b00ce4b3cfab14a648b2d"
server_key_version: "encryptionkey@0"
I20260924 11:45:24.124625 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:45:24.127733 19112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:24.128983 19098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:45:24.129151 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "907f47e3d69d414bad230c3a9e7d49be"
format_stamp: "Formatted at 2026-09-24 11:45:24 on dist-test-slave-qhrh"
server_key: "c51196f6a872d8fed63246850767b09b"
server_key_iv: "d65f5f32d26b00ce4b3cfab14a648b2d"
server_key_version: "encryptionkey@0"
I20260924 11:45:24.129355 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:24.164011 19098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:24.164888 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:24.165177 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:24.183137 19098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34395
I20260924 11:45:24.183175 19163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34395 every 8 connection(s)
I20260924 11:45:24.184446 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:24.187887 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:24.188781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19098
I20260924 11:45:24.188931 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:24.189343 388 external_mini_cluster.cc:1428] Setting key ef3bbcdc8258f2d4fc186caf2d4d9ab1
I20260924 11:45:24.199302 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be: Bootstrap starting.
I20260924 11:45:24.201660 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:24.191155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.0.97.62:34395)
0924 11:45:24.192075 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:24.192085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:24.192130 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:45:24.193900 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:24.193919 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:24.194323 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:24.194666 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:24.195135 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:24.196284 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:24.197508 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:24.198094 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:24.198281 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:24.199454 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:24.199464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:24.199480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:24.199998 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:24.200065 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:24.200095 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:24.200279 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:24.200479 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 11:45:24.200767 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":330,"threads_started":1}
I20260924 11:45:24.202672 19164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:24.203658 19164 log.cc:824] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:24.206312 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be: No bootstrap required, opened a new log
I20260924 11:45:24.213135 19164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907f47e3d69d414bad230c3a9e7d49be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34395 } }
I20260924 11:45:24.213462 19164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:24.213563 19164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907f47e3d69d414bad230c3a9e7d49be, State: Initialized, Role: FOLLOWER
I20260924 11:45:24.214046 19164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907f47e3d69d414bad230c3a9e7d49be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34395 } }
I20260924 11:45:24.214283 19164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:24.214414 19164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:24.214567 19164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:24.216825 19164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907f47e3d69d414bad230c3a9e7d49be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34395 } }
I20260924 11:45:24.217279 19164 leader_election.cc:302] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907f47e3d69d414bad230c3a9e7d49be; no voters:
I20260924 11:45:24.217797 19164 leader_election.cc:288] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:24.217914 19169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:24.218569 19169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [term 1 LEADER]: Becoming Leader. State: Replica: 907f47e3d69d414bad230c3a9e7d49be, State: Running, Role: LEADER
I20260924 11:45:24.219007 19169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907f47e3d69d414bad230c3a9e7d49be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34395 } }
I20260924 11:45:24.219652 19164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:24.221869 19171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907f47e3d69d414bad230c3a9e7d49be. Latest consensus state: current_term: 1 leader_uuid: "907f47e3d69d414bad230c3a9e7d49be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907f47e3d69d414bad230c3a9e7d49be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34395 } } }
I20260924 11:45:24.221776 19170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907f47e3d69d414bad230c3a9e7d49be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907f47e3d69d414bad230c3a9e7d49be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34395 } } }
I20260924 11:45:24.222231 19171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:24.222235 19170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:24.229333 19176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:24.235366 19176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:24.244850 19176 catalog_manager.cc:1347] Generated new cluster ID: 44d5f3a13e044c529c2d759c647d3ec7
I20260924 11:45:24.244942 19176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:24.259578 19176 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:24.260829 19176 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:24.266974 19176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be: Generated new TSK 0
I20260924 11:45:24.267747 19176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:24.279503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34395
--builtin_ntp_servers=127.0.97.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:45:24.594297 19188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:24.594532 19188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:24.594607 19188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:24.601150 19188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:24.601238 19188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:24.601338 19188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:45:24.609926 19188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:24.611019 19188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:24.612774 19188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:24.625402 19193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:24.625408 19194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:24.625432 19196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:24.626250 19188 server_base.cc:1034] running on GCE node
I20260924 11:45:24.626878 19188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:24.627765 19188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:24.628980 19188 hybrid_clock.cc:648] HybridClock initialized: now 1790250324628930 us; error 37 us; skew 500 ppm
I20260924 11:45:24.629280 19188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:24.631330 19188 webserver.cc:460] Webserver started at http://127.0.97.1:34915/ using document root <none> and password file <none>
I20260924 11:45:24.631981 19188 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:24.632125 19188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:24.632480 19188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:24.635212 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "c411b84500b240a180771c33787798a0"
format_stamp: "Formatted at 2026-09-24 11:45:24 on dist-test-slave-qhrh"
server_key: "14a28cb62ea08ae40c6b20664a82a909"
server_key_iv: "0272199bf2ea2f6252dd18c2a6d22952"
server_key_version: "encryptionkey@0"
I20260924 11:45:24.635963 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c411b84500b240a180771c33787798a0"
format_stamp: "Formatted at 2026-09-24 11:45:24 on dist-test-slave-qhrh"
server_key: "14a28cb62ea08ae40c6b20664a82a909"
server_key_iv: "0272199bf2ea2f6252dd18c2a6d22952"
server_key_version: "encryptionkey@0"
I20260924 11:45:24.640458 19188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:45:24.643570 19202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:24.644871 19188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:45:24.645053 19188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "c411b84500b240a180771c33787798a0"
format_stamp: "Formatted at 2026-09-24 11:45:24 on dist-test-slave-qhrh"
server_key: "14a28cb62ea08ae40c6b20664a82a909"
server_key_iv: "0272199bf2ea2f6252dd18c2a6d22952"
server_key_version: "encryptionkey@0"
I20260924 11:45:24.645253 19188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:24.657802 19188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:24.658532 19188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:24.658795 19188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:24.659616 19188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:24.661074 19188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:24.661191 19188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:24.661329 19188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:24.661418 19188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:24.699810 19188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43587
I20260924 11:45:24.699896 19314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43587 every 8 connection(s)
I20260924 11:45:24.701112 19188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:45:24.705466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19188
I20260924 11:45:24.705626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:45:24.706030 388 external_mini_cluster.cc:1428] Setting key 3e88a69c048aa0ce26410a4c60a88323
I20260924 11:45:24.708508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34395
--builtin_ntp_servers=127.0.97.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:45:24.713451 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:24.703740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50973 (local address 127.0.97.62:34395)
0924 11:45:24.704268 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:24.704273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:24.705359 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 11:45:24.707183 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:24.707188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:24.707294 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:24.707402 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:24.708423 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:24.709222 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:24.710695 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:24.711104 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:24.711245 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:24.712624 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:24.712630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:24.712635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:24.712663 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:24.712699 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:24.712712 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:24.712809 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:24.713044 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 11:45:24.713129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
I20260924 11:45:24.714340 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:24.704266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34395 (local address 127.0.97.1:50973)
0924 11:45:24.705220 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:24.705260 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:24.706938 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:24.707611 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:24.707618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:24.707694 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:24.708280 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:24.708297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:24.709371 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:24.709376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:24.710558 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:24.710569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:24.711344 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:24.711353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:24.711573 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:24.712327 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:24.712330 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:24.712368 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:24.712436 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:24.712807 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:24.712819 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:24.712829 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:45:24.713005 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 11:45:24.713377 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":260,"threads_started":1}
I20260924 11:45:24.715852 19315 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34395
I20260924 11:45:24.716185 19315 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:24.717048 19315 heartbeater.cc:511] Master 127.0.97.62:34395 requested a full tablet report, sending...
I20260924 11:45:24.719044 19129 ts_manager.cc:194] Registered new tserver with Master: c411b84500b240a180771c33787798a0 (127.0.97.1:43587)
I20260924 11:45:24.720662 19129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50973
W20260924 11:45:25.017138 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:25.017437 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:25.017578 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:25.024585 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:25.024691 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:25.024856 19319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:45:25.033674 19319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:25.034832 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:25.036619 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:25.049554 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:25.049515 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:25.049598 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:25.053565 19319 server_base.cc:1034] running on GCE node
I20260924 11:45:25.054127 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:25.054994 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:25.056237 19319 hybrid_clock.cc:648] HybridClock initialized: now 1790250325056193 us; error 50 us; skew 500 ppm
I20260924 11:45:25.056526 19319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:25.058559 19319 webserver.cc:460] Webserver started at http://127.0.97.2:34029/ using document root <none> and password file <none>
I20260924 11:45:25.059203 19319 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:25.059327 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:25.059607 19319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:25.062429 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "dd6657fdfeb2450695a033c7cb39d676"
format_stamp: "Formatted at 2026-09-24 11:45:25 on dist-test-slave-qhrh"
server_key: "c13dcb99758e98a6656cb0fc9f82c1af"
server_key_iv: "784f417f766220cdd53a791aa348afc8"
server_key_version: "encryptionkey@0"
I20260924 11:45:25.063191 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dd6657fdfeb2450695a033c7cb39d676"
format_stamp: "Formatted at 2026-09-24 11:45:25 on dist-test-slave-qhrh"
server_key: "c13dcb99758e98a6656cb0fc9f82c1af"
server_key_iv: "784f417f766220cdd53a791aa348afc8"
server_key_version: "encryptionkey@0"
I20260924 11:45:25.067742 19319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260924 11:45:25.070878 19333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:25.072139 19319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:25.072372 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "dd6657fdfeb2450695a033c7cb39d676"
format_stamp: "Formatted at 2026-09-24 11:45:25 on dist-test-slave-qhrh"
server_key: "c13dcb99758e98a6656cb0fc9f82c1af"
server_key_iv: "784f417f766220cdd53a791aa348afc8"
server_key_version: "encryptionkey@0"
I20260924 11:45:25.072582 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:25.089854 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:25.090533 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:25.090785 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:25.091606 19319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:25.093006 19319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:25.093144 19319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:25.093274 19319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:25.093360 19319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:25.125101 19319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43673
I20260924 11:45:25.125185 19445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43673 every 8 connection(s)
I20260924 11:45:25.126375 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:45:25.132618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19319
I20260924 11:45:25.132814 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:45:25.133280 388 external_mini_cluster.cc:1428] Setting key eb17e1b35fa4b28c4f469ad6b5a8eb85
I20260924 11:45:25.135743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34395
--builtin_ntp_servers=127.0.97.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:45:25.140245 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.128924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56477 (local address 127.0.97.62:34395)
0924 11:45:25.129113 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.129117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.130450 (+ 1333us) server_negotiation.cc:407] Connection header received
0924 11:45:25.132230 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.132236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.132325 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.132421 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:25.134211 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.135079 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.136991 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.137690 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.137840 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.139318 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:25.139327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:25.139334 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:25.139356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:25.139395 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:25.139413 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:25.139503 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.139758 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.139858 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:45:25.141142 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.129442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34395 (local address 127.0.97.2:56477)
0924 11:45:25.130302 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.130336 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.131952 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.133113 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.133122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.133260 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:25.134018 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.134045 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.135257 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.135263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.136827 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.136840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.137911 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.137916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.138157 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.138952 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:25.138954 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:25.138992 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:25.139069 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:25.139514 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:25.139525 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:25.139535 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.139714 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.140067 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260924 11:45:25.142436 19446 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34395
I20260924 11:45:25.142758 19446 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:25.143558 19446 heartbeater.cc:511] Master 127.0.97.62:34395 requested a full tablet report, sending...
I20260924 11:45:25.145053 19129 ts_manager.cc:194] Registered new tserver with Master: dd6657fdfeb2450695a033c7cb39d676 (127.0.97.2:43673)
I20260924 11:45:25.146333 19129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:56477
W20260924 11:45:25.452953 19449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:25.453236 19449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:25.453318 19449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:25.460245 19449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:25.460351 19449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:25.460482 19449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:45:25.468945 19449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:25.470161 19449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:25.471787 19449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:25.484406 19457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:25.484434 19454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:25.484362 19455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:25.485168 19449 server_base.cc:1034] running on GCE node
I20260924 11:45:25.485756 19449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:25.486634 19449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:25.487848 19449 hybrid_clock.cc:648] HybridClock initialized: now 1790250325487816 us; error 48 us; skew 500 ppm
I20260924 11:45:25.488181 19449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:25.490193 19449 webserver.cc:460] Webserver started at http://127.0.97.3:40525/ using document root <none> and password file <none>
I20260924 11:45:25.490863 19449 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:25.490993 19449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:25.491305 19449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:25.493997 19449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "6b24d170d60241a58e93f6678206b5e3"
format_stamp: "Formatted at 2026-09-24 11:45:25 on dist-test-slave-qhrh"
server_key: "a9f121ef21dd6f3f04f2bfb6524da936"
server_key_iv: "cbafb989b50beeb14e3101908d121610"
server_key_version: "encryptionkey@0"
I20260924 11:45:25.494742 19449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6b24d170d60241a58e93f6678206b5e3"
format_stamp: "Formatted at 2026-09-24 11:45:25 on dist-test-slave-qhrh"
server_key: "a9f121ef21dd6f3f04f2bfb6524da936"
server_key_iv: "cbafb989b50beeb14e3101908d121610"
server_key_version: "encryptionkey@0"
I20260924 11:45:25.499106 19449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:45:25.502329 19463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:25.503509 19449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:25.503667 19449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "6b24d170d60241a58e93f6678206b5e3"
format_stamp: "Formatted at 2026-09-24 11:45:25 on dist-test-slave-qhrh"
server_key: "a9f121ef21dd6f3f04f2bfb6524da936"
server_key_iv: "cbafb989b50beeb14e3101908d121610"
server_key_version: "encryptionkey@0"
I20260924 11:45:25.503865 19449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:25.533033 19449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:25.533794 19449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:25.534066 19449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:25.534921 19449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:25.536206 19449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:25.536319 19449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:25.536429 19449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:25.536515 19449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:25.567484 19449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37695
I20260924 11:45:25.567565 19575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37695 every 8 connection(s)
I20260924 11:45:25.568711 19449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:45:25.571005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19449
I20260924 11:45:25.571142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:45:25.571635 388 external_mini_cluster.cc:1428] Setting key 83db0bc50bf745152ed8959c7867831c
I20260924 11:45:25.580549 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.571208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42367 (local address 127.0.97.62:34395)
0924 11:45:25.571450 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.571458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.573125 (+ 1667us) server_negotiation.cc:407] Connection header received
0924 11:45:25.575150 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.575154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.575214 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.575309 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:25.576341 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.577102 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.578415 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.578765 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.578843 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.579848 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:25.579852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:25.579855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:25.579869 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:25.579896 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:25.579906 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:25.579966 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.580123 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.580235 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:45:25.581455 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.571752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34395 (local address 127.0.97.3:42367)
0924 11:45:25.572951 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.573003 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.574921 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.575526 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.575541 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.575619 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:25.576127 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.576187 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.577209 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.577213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.578296 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.578306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.578911 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.578914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.579073 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.579615 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:25.579617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:25.579649 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:25.579708 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:25.579977 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:25.579983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:25.579987 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.580101 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.580467 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":469,"threads_started":1}
I20260924 11:45:25.582656 19576 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34395
I20260924 11:45:25.582929 19576 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:25.583565 19576 heartbeater.cc:511] Master 127.0.97.62:34395 requested a full tablet report, sending...
I20260924 11:45:25.584767 19129 ts_manager.cc:194] Registered new tserver with Master: 6b24d170d60241a58e93f6678206b5e3 (127.0.97.3:37695)
I20260924 11:45:25.585870 19129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:42367
I20260924 11:45:25.586460 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:45:25.594980 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.589972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.0.97.62:34395)
0924 11:45:25.590213 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.590218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.590486 (+ 268us) server_negotiation.cc:407] Connection header received
0924 11:45:25.590593 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.590597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.590654 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.590724 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:25.591119 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.591703 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.592986 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.593333 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.593410 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.594300 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:25.594303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:25.594306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:25.594319 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:25.594342 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:25.594351 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:25.594470 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.594639 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.594700 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:45:25.598031 19129 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:45:25.604331 19129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:45:25.632465 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.615194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37695 (local address 127.0.0.1:33830)
0924 11:45:25.616256 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.616298 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.616658 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.618189 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.618199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.618236 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.619025 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.619058 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.625071 (+ 6013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.625080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.628186 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.628206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.631396 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.631403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.631665 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.631744 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.631866 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.632067 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":1,"thread_start_us":1380,"threads_started":1}
I20260924 11:45:25.632471 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.615413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.0.97.3:37695)
0924 11:45:25.617450 (+ 2037us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.617457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.617499 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:45:25.617618 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.617629 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.617926 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.620625 (+ 2699us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.621524 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.624893 (+ 3369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.629910 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.631207 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.631355 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.631641 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.631962 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.632127 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":1719,"threads_started":1}
I20260924 11:45:25.636560 19511 tablet_service.cc:1467] Processing CreateTablet for tablet 985a05300e9e452585aa3d801c3a3dd8 (DEFAULT_TABLE table=test-table [id=b0b9d95c9b674defa361a339c7ff1dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:25.637193 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.616065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43673 (local address 127.0.0.1:40420)
0924 11:45:25.629632 (+ 13567us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.629647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.629792 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.630461 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.630467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.630492 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.630786 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.630798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.632765 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.632770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.634881 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.634897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.636431 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.636436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.636627 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.636666 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.636754 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.636868 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13436,"mutex_wait_us":47,"thread_start_us":13294,"threads_started":1}
I20260924 11:45:25.637506 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.616500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.97.2:43673)
0924 11:45:25.629198 (+ 12698us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.629220 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.629749 (+ 529us) server_negotiation.cc:407] Connection header received
0924 11:45:25.629956 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.629967 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.630207 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.630398 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.630973 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.632627 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.635105 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.636269 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.636406 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.636673 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.636796 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.636934 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12522,"thread_start_us":12410,"threads_started":1}
I20260924 11:45:25.638135 19511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985a05300e9e452585aa3d801c3a3dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:25.638351 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.617300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58598)
0924 11:45:25.630236 (+ 12936us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.630254 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.630375 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.631190 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.631194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.631221 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.631618 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.631629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.633529 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.633535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.635562 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.635575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.637651 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.637660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.637810 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.637830 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.637921 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.638021 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12833,"mutex_wait_us":1,"thread_start_us":12720,"threads_started":1}
I20260924 11:45:25.638767 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.617698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.0.97.1:43587)
0924 11:45:25.628613 (+ 10915us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.628624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.630366 (+ 1742us) server_negotiation.cc:407] Connection header received
0924 11:45:25.630583 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.630590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.630773 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.630914 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.631789 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.633400 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.635773 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.637130 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.637257 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.637616 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.637975 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.638127 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10711,"thread_start_us":10895,"threads_started":1}
I20260924 11:45:25.640866 19381 tablet_service.cc:1467] Processing CreateTablet for tablet 985a05300e9e452585aa3d801c3a3dd8 (DEFAULT_TABLE table=test-table [id=b0b9d95c9b674defa361a339c7ff1dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:25.641492 19250 tablet_service.cc:1467] Processing CreateTablet for tablet 985a05300e9e452585aa3d801c3a3dd8 (DEFAULT_TABLE table=test-table [id=b0b9d95c9b674defa361a339c7ff1dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:25.642282 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985a05300e9e452585aa3d801c3a3dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:25.642638 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985a05300e9e452585aa3d801c3a3dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:25.652057 19594 tablet_bootstrap.cc:492] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Bootstrap starting.
I20260924 11:45:25.653811 19595 tablet_bootstrap.cc:492] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Bootstrap starting.
I20260924 11:45:25.655411 19596 tablet_bootstrap.cc:492] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Bootstrap starting.
I20260924 11:45:25.657013 19595 tablet_bootstrap.cc:654] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:25.657013 19594 tablet_bootstrap.cc:654] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:25.657789 19596 tablet_bootstrap.cc:654] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:25.658131 19595 log.cc:824] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:25.658131 19594 log.cc:824] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:25.658784 19596 log.cc:824] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:25.660845 19595 tablet_bootstrap.cc:492] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: No bootstrap required, opened a new log
I20260924 11:45:25.661154 19595 ts_tablet_manager.cc:1397] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:45:25.661693 19594 tablet_bootstrap.cc:492] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: No bootstrap required, opened a new log
I20260924 11:45:25.661691 19596 tablet_bootstrap.cc:492] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: No bootstrap required, opened a new log
I20260924 11:45:25.661938 19596 ts_tablet_manager.cc:1397] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 11:45:25.661940 19594 ts_tablet_manager.cc:1397] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260924 11:45:25.668084 19595 raft_consensus.cc:348] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.668471 19595 raft_consensus.cc:374] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:25.668593 19595 raft_consensus.cc:729] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c411b84500b240a180771c33787798a0, State: Initialized, Role: FOLLOWER
I20260924 11:45:25.669108 19595 consensus_queue.cc:260] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.670095 19315 heartbeater.cc:503] Master 127.0.97.62:34395 was elected leader, sending a full tablet report...
I20260924 11:45:25.670722 19595 ts_tablet_manager.cc:1428] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260924 11:45:25.672060 19596 raft_consensus.cc:348] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.672067 19594 raft_consensus.cc:348] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.672482 19594 raft_consensus.cc:374] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:25.672482 19596 raft_consensus.cc:374] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:25.672590 19596 raft_consensus.cc:729] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6657fdfeb2450695a033c7cb39d676, State: Initialized, Role: FOLLOWER
I20260924 11:45:25.672590 19594 raft_consensus.cc:729] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b24d170d60241a58e93f6678206b5e3, State: Initialized, Role: FOLLOWER
I20260924 11:45:25.673130 19594 consensus_queue.cc:260] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.673130 19596 consensus_queue.cc:260] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.674721 19576 heartbeater.cc:503] Master 127.0.97.62:34395 was elected leader, sending a full tablet report...
I20260924 11:45:25.674930 19596 ts_tablet_manager.cc:1428] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 11:45:25.675222 19594 ts_tablet_manager.cc:1428] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 11:45:25.676077 19446 heartbeater.cc:503] Master 127.0.97.62:34395 was elected leader, sending a full tablet report...
W20260924 11:45:25.703210 19316 tablet.cc:2369] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:25.807317 19602 raft_consensus.cc:482] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:25.807626 19602 raft_consensus.cc:504] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.808211 19601 raft_consensus.cc:482] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:25.808563 19601 raft_consensus.cc:504] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.809515 19602 leader_election.cc:288] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c411b84500b240a180771c33787798a0 (127.0.97.1:43587), dd6657fdfeb2450695a033c7cb39d676 (127.0.97.2:43673)
I20260924 11:45:25.810220 19601 leader_election.cc:288] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b24d170d60241a58e93f6678206b5e3 (127.0.97.3:37695), c411b84500b240a180771c33787798a0 (127.0.97.1:43587)
I20260924 11:45:25.817332 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.809893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.97.3:55763)
0924 11:45:25.810361 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.810386 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.810530 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.811055 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.811060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.811089 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.811300 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.811315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.812778 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.812784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.814870 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.814883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.816551 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.816556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.816766 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.816797 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.816884 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.817016 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":2}
I20260924 11:45:25.817953 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.810505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55763 (local address 127.0.97.1:43587)
0924 11:45:25.810681 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.810687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.810707 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:45:25.810794 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.810799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.810883 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.810968 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.811499 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.812616 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.815053 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.816252 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.816341 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.816435 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.817587 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.817710 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 11:45:25.818849 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a05300e9e452585aa3d801c3a3dd8" candidate_uuid: "6b24d170d60241a58e93f6678206b5e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411b84500b240a180771c33787798a0" is_pre_election: true
I20260924 11:45:25.819590 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.810870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58949 (local address 127.0.97.3:37695)
0924 11:45:25.811014 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.811020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.811848 (+ 828us) server_negotiation.cc:407] Connection header received
0924 11:45:25.812097 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.812102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.812189 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.812255 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.813794 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.814888 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.817296 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.818387 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.818485 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.818571 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.819273 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.819389 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":30}
I20260924 11:45:25.819586 19270 raft_consensus.cc:2454] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b24d170d60241a58e93f6678206b5e3 in term 0.
I20260924 11:45:25.820282 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.810567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37695 (local address 127.0.97.2:58949)
0924 11:45:25.811761 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.811778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.811942 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.812435 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.812439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.812462 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.813637 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.813653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.815057 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.815061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.817084 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.817099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.818744 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.818750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.819076 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.819115 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.819783 (+ 668us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.819909 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":775,"thread_start_us":853,"threads_started":1}
W20260924 11:45:25.820336 19577 tablet.cc:2369] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:25.821226 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.809946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43479 (local address 127.0.97.2:43673)
0924 11:45:25.810153 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.810160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.811135 (+ 975us) server_negotiation.cc:407] Connection header received
0924 11:45:25.811462 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.811470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.811548 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.811619 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.812534 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.813961 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.817180 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.818824 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.818990 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.819573 (+ 583us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.820689 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.820863 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":429,"server-negotiator.queue_time_us":88}
I20260924 11:45:25.821492 19464 leader_election.cc:302] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b24d170d60241a58e93f6678206b5e3, c411b84500b240a180771c33787798a0; no voters:
I20260924 11:45:25.822329 19602 raft_consensus.cc:2781] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:45:25.822471 19602 raft_consensus.cc:482] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:45:25.822664 19602 raft_consensus.cc:3037] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:25.822398 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.809937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43673 (local address 127.0.97.3:43479)
0924 11:45:25.811024 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.811043 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.811279 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.811824 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.811828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.811851 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.812304 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.812321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.814185 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.814193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.816927 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.816947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.819927 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.819935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.820188 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.820233 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.821642 (+ 1409us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.821890 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":2,"thread_start_us":1103,"threads_started":1}
I20260924 11:45:25.824349 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a05300e9e452585aa3d801c3a3dd8" candidate_uuid: "dd6657fdfeb2450695a033c7cb39d676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b24d170d60241a58e93f6678206b5e3" is_pre_election: true
I20260924 11:45:25.824678 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a05300e9e452585aa3d801c3a3dd8" candidate_uuid: "6b24d170d60241a58e93f6678206b5e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6657fdfeb2450695a033c7cb39d676" is_pre_election: true
I20260924 11:45:25.825078 19401 raft_consensus.cc:2454] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b24d170d60241a58e93f6678206b5e3 in term 0.
I20260924 11:45:25.826081 19602 raft_consensus.cc:504] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.826522 19531 raft_consensus.cc:2379] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd6657fdfeb2450695a033c7cb39d676 in current term 1: Already voted for candidate 6b24d170d60241a58e93f6678206b5e3 in this term.
I20260924 11:45:25.827090 19602 leader_election.cc:288] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [CANDIDATE]: Term 1 election: Requested vote from peers c411b84500b240a180771c33787798a0 (127.0.97.1:43587), dd6657fdfeb2450695a033c7cb39d676 (127.0.97.2:43673)
I20260924 11:45:25.827200 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a05300e9e452585aa3d801c3a3dd8" candidate_uuid: "6b24d170d60241a58e93f6678206b5e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411b84500b240a180771c33787798a0"
I20260924 11:45:25.827418 19270 raft_consensus.cc:3037] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:25.827418 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a05300e9e452585aa3d801c3a3dd8" candidate_uuid: "6b24d170d60241a58e93f6678206b5e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6657fdfeb2450695a033c7cb39d676"
I20260924 11:45:25.827598 19401 raft_consensus.cc:3037] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:25.828022 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.811809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.97.2:34093)
0924 11:45:25.820502 (+ 8693us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:25.820519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:25.820686 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:25.821276 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:25.821280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:25.821307 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:25.821457 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.821471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.822992 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.822999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.825468 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:25.825480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.827248 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.827254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:25.827449 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.827480 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:45:25.827565 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 11:45:25.827703 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8599,"thread_start_us":11641,"threads_started":1}
I20260924 11:45:25.828691 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.812195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34093 (local address 127.0.97.1:43587)
0924 11:45:25.813280 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.813286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.820597 (+ 7311us) server_negotiation.cc:407] Connection header received
0924 11:45:25.820825 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.820830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.820895 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.820980 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:25.821661 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.822865 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.825667 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.826917 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.827033 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.827157 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.828217 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.828342 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":712,"threads_started":1}
I20260924 11:45:25.829239 19269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a05300e9e452585aa3d801c3a3dd8" candidate_uuid: "dd6657fdfeb2450695a033c7cb39d676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411b84500b240a180771c33787798a0" is_pre_election: true
I20260924 11:45:25.830027 19270 raft_consensus.cc:2454] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b24d170d60241a58e93f6678206b5e3 in term 1.
I20260924 11:45:25.830029 19334 leader_election.cc:302] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd6657fdfeb2450695a033c7cb39d676; no voters: 6b24d170d60241a58e93f6678206b5e3, c411b84500b240a180771c33787798a0
I20260924 11:45:25.830430 19464 leader_election.cc:302] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b24d170d60241a58e93f6678206b5e3, c411b84500b240a180771c33787798a0; no voters:
I20260924 11:45:25.830659 19401 raft_consensus.cc:2454] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b24d170d60241a58e93f6678206b5e3 in term 1.
I20260924 11:45:25.830691 19602 raft_consensus.cc:2781] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:25.830829 19601 raft_consensus.cc:2726] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 11:45:25.831475 19602 raft_consensus.cc:686] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 1 LEADER]: Becoming Leader. State: Replica: 6b24d170d60241a58e93f6678206b5e3, State: Running, Role: LEADER
I20260924 11:45:25.831915 19602 consensus_queue.cc:237] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } }
I20260924 11:45:25.835963 19128 catalog_manager.cc:5495] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b24d170d60241a58e93f6678206b5e3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6b24d170d60241a58e93f6678206b5e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b24d170d60241a58e93f6678206b5e3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 } health_report { overall_health: UNKNOWN } } }
I20260924 11:45:25.861804 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.856856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.0.97.3:37695)
0924 11:45:25.857009 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.857013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.857029 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:45:25.857211 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.857215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.857342 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.857455 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:45:25.857789 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.858533 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.859916 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.860365 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.860452 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.860886 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:45:25.861342 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:45:25.861461 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.861566 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.861598 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:25.869412 19270 raft_consensus.cc:1261] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 1 FOLLOWER]: Refusing update from remote peer 6b24d170d60241a58e93f6678206b5e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:25.869468 19401 raft_consensus.cc:1261] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 1 FOLLOWER]: Refusing update from remote peer 6b24d170d60241a58e93f6678206b5e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:25.870294 19607 consensus_queue.cc:1035] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6657fdfeb2450695a033c7cb39d676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:25.870642 19602 consensus_queue.cc:1035] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c411b84500b240a180771c33787798a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:45:25.878890 19447 tablet.cc:2369] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:25.881028 19610 mvcc.cc:205] Tried to move back new op lower bound from 7332865334751248384 to 7332865334611599360. Current Snapshot: MvccSnapshot[applied={T|T < 7332865334751248384}]
I20260924 11:45:25.896255 19129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:25.896487 19129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:25.899907 19129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 907f47e3d69d414bad230c3a9e7d49be: Sending DeleteTablet for 3 replicas of tablet 985a05300e9e452585aa3d801c3a3dd8
I20260924 11:45:25.901017 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a05300e9e452585aa3d801c3a3dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:25 UTC) from {username='slave'} at 127.0.0.1:40420
I20260924 11:45:25.901044 19511 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a05300e9e452585aa3d801c3a3dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:25 UTC) from {username='slave'} at 127.0.0.1:33830
I20260924 11:45:25.901137 19250 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a05300e9e452585aa3d801c3a3dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:25 UTC) from {username='slave'} at 127.0.0.1:58598
I20260924 11:45:25.901904 19620 tablet_replica.cc:330] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: stopping tablet replica
I20260924 11:45:25.902347 19620 raft_consensus.cc:2229] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:45:25.902782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19188
I20260924 11:45:25.903069 19620 raft_consensus.cc:2258] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:25.905043 19620 ts_tablet_manager.cc:1905] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:25.912276 19620 ts_tablet_manager.cc:1918] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:25.912487 19620 log.cc:1196] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/985a05300e9e452585aa3d801c3a3dd8
I20260924 11:45:25.913117 19620 ts_tablet_manager.cc:1939] T 985a05300e9e452585aa3d801c3a3dd8 P 6b24d170d60241a58e93f6678206b5e3: Deleting consensus metadata
I20260924 11:45:25.914789 19113 catalog_manager.cc:4841] TS 6b24d170d60241a58e93f6678206b5e3 (127.0.97.3:37695): tablet 985a05300e9e452585aa3d801c3a3dd8 (table test-table [id=b0b9d95c9b674defa361a339c7ff1dcb]) successfully deleted
I20260924 11:45:25.916657 19622 tablet_replica.cc:330] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: stopping tablet replica
I20260924 11:45:25.917137 19622 raft_consensus.cc:2229] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:25.917528 19621 tablet_replica.cc:330] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: stopping tablet replica
I20260924 11:45:25.917810 19622 raft_consensus.cc:2258] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:25.917912 19621 raft_consensus.cc:2229] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:25.918360 19621 raft_consensus.cc:2258] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:25.920507 19621 ts_tablet_manager.cc:1905] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:25.920979 19622 ts_tablet_manager.cc:1905] T 985a05300e9e452585aa3d801c3a3dd8 P c411b84500b240a180771c33787798a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:25.926468 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:25.905234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.0.97.1:43587)
0924 11:45:25.905447 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:25.905455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:25.916882 (+ 11427us) server_negotiation.cc:407] Connection header received
0924 11:45:25.917075 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:25.917083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:25.917249 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:25.917386 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:25.918021 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.919206 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.921694 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:25.922358 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:25.922452 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:25.924201 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:25.924214 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:25.924229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:25.925244 (+ 1015us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:25.925324 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:25.925354 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:25.925537 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:25.925885 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 11:45:25.926010 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:45:25.927460 19310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:25.927512 19621 ts_tablet_manager.cc:1918] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:25.927695 19621 log.cc:1196] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/985a05300e9e452585aa3d801c3a3dd8
I20260924 11:45:25.928181 19621 ts_tablet_manager.cc:1939] T 985a05300e9e452585aa3d801c3a3dd8 P dd6657fdfeb2450695a033c7cb39d676: Deleting consensus metadata
I20260924 11:45:25.929495 19114 catalog_manager.cc:4841] TS dd6657fdfeb2450695a033c7cb39d676 (127.0.97.2:43673): tablet 985a05300e9e452585aa3d801c3a3dd8 (table test-table [id=b0b9d95c9b674defa361a339c7ff1dcb]) successfully deleted
I20260924 11:45:26.017813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19188
I20260924 11:45:26.041556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19319
W20260924 11:45:26.041479 19113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43587 (error 108)
I20260924 11:45:26.043440 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.042938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58622)
0924 11:45:26.043123 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.043265 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:45:26.043795 19113 catalog_manager.cc:4584] TS c411b84500b240a180771c33787798a0 (127.0.97.1:43587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 985a05300e9e452585aa3d801c3a3dd8: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
I20260924 11:45:26.048938 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.043030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.97.2:43673)
0924 11:45:26.043206 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:26.043212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:26.043229 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:45:26.043305 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:26.043309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:26.043443 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:26.043546 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:26.044027 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:26.044799 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:26.046002 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:26.046394 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:26.046531 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:26.047607 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:26.047615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:26.047628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:26.048214 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:26.048264 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:26.048284 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:26.048408 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:26.048531 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:45:26.048630 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:45:26.049536 19441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:26.097429 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.096907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58628)
0924 11:45:26.097121 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.097222 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 11:45:26.098505 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.098148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58636)
0924 11:45:26.098294 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.098373 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:45:26.137454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19319
I20260924 11:45:26.154635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19449
I20260924 11:45:26.158687 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.158220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58648)
0924 11:45:26.158423 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.158526 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 11:45:26.159701 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.159355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58652)
0924 11:45:26.159499 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.159584 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 11:45:26.162348 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.155431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.0.97.3:37695)
0924 11:45:26.155639 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:26.155650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:26.155672 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:45:26.155766 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:26.155782 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:26.155906 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:26.156022 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:26.156519 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:26.157219 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:26.158550 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:26.159243 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:26.159399 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:26.160705 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:26.160715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:26.160736 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:26.161580 (+ 844us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:26.161644 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:26.161667 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:26.161793 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:26.161907 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:45:26.162008 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 11:45:26.163100 19571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:26.240453 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.239905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58658)
0924 11:45:26.240121 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.240267 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 11:45:26.241497 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:26.241092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:58672)
0924 11:45:26.241291 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:26.241370 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 11:45:26.285255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19449
I20260924 11:45:26.303898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19098
I20260924 11:45:26.304963 19159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:26.407202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19098
2026-09-24T11:45:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2706 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:26 dist-test-slave-qhrh krb5kdc[19631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:26 dist-test-slave-qhrh krb5kdc[19631](info): set up 2 sockets
Sep 24 11:45:26 dist-test-slave-qhrh krb5kdc[19631](info): commencing operation
krb5kdc: starting...
W20260924 11:45:27.452096 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:27 dist-test-slave-qhrh krb5kdc[19631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:45:27.667248 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42293
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:27.973801 19644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:27.974084 19644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:27.974174 19644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:27.980521 19644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:27.980609 19644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:27.980649 19644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:27.980700 19644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:27.980738 19644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:27.989229 19644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:27.990383 19644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:27.991586 19644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:28.002408 19650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:28.002537 19649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:28.002470 19652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:28.003366 19644 server_base.cc:1034] running on GCE node
I20260924 11:45:28.004092 19644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:28.005355 19644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:28.006561 19644 hybrid_clock.cc:648] HybridClock initialized: now 1790250328006524 us; error 28 us; skew 500 ppm
I20260924 11:45:28.006978 19644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:28 dist-test-slave-qhrh krb5kdc[19631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:28.023504 19644 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:45:28.025118 19644 webserver.cc:460] Webserver started at http://127.0.97.62:42947/ using document root <none> and password file <none>
I20260924 11:45:28.025802 19644 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:28.025938 19644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:28.026266 19644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:28.029120 19644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "b724003641a645388f53f4a3ebd13a8b"
format_stamp: "Formatted at 2026-09-24 11:45:28 on dist-test-slave-qhrh"
server_key: "5a8a55bb5605e9781aa8515330c184a1"
server_key_iv: "ff8994ebc575b67ad8f91f0a8f73e951"
server_key_version: "encryptionkey@0"
I20260924 11:45:28.029837 19644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "b724003641a645388f53f4a3ebd13a8b"
format_stamp: "Formatted at 2026-09-24 11:45:28 on dist-test-slave-qhrh"
server_key: "5a8a55bb5605e9781aa8515330c184a1"
server_key_iv: "ff8994ebc575b67ad8f91f0a8f73e951"
server_key_version: "encryptionkey@0"
I20260924 11:45:28.034713 19644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:28.037971 19659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:28.039141 19644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:28.039311 19644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "b724003641a645388f53f4a3ebd13a8b"
format_stamp: "Formatted at 2026-09-24 11:45:28 on dist-test-slave-qhrh"
server_key: "5a8a55bb5605e9781aa8515330c184a1"
server_key_iv: "ff8994ebc575b67ad8f91f0a8f73e951"
server_key_version: "encryptionkey@0"
I20260924 11:45:28.039489 19644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:28.051539 19644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:28.055133 19644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:28.055392 19644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:28.072649 19644 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42293
I20260924 11:45:28.072669 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42293 every 8 connection(s)
I20260924 11:45:28.073995 19644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:28.077441 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:28.078017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19644
I20260924 11:45:28.078166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:28.078691 388 external_mini_cluster.cc:1428] Setting key 70a07f917c2fc35230827b791aebae8b
I20260924 11:45:28.090373 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b: Bootstrap starting.
Sep 24 11:45:28 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:28.093834 19711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:28.094877 19711 log.cc:824] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:28.097620 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b: No bootstrap required, opened a new log
I20260924 11:45:28.100674 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:28.080445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.0.97.62:42293)
0924 11:45:28.081134 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:28.081145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:28.081181 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:45:28.083200 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:28.083221 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:28.083607 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:28.083909 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:28.085596 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:28.086799 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:28.088546 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:28.089141 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:28.089269 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:28.093807 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:28.093814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:28.093830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:28.093864 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:28.097025 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:28.097628 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:28.097633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:28.097638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:28.097826 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:28.098175 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:28.098180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:28.098183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:28.098626 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:28.098926 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:28.099293 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 11:45:28.099665 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260924 11:45:28.105062 19711 raft_consensus.cc:348] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b724003641a645388f53f4a3ebd13a8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260924 11:45:28.105414 19711 raft_consensus.cc:374] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:28.105496 19711 raft_consensus.cc:729] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b724003641a645388f53f4a3ebd13a8b, State: Initialized, Role: FOLLOWER
I20260924 11:45:28.105954 19711 consensus_queue.cc:260] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b724003641a645388f53f4a3ebd13a8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260924 11:45:28.106144 19711 raft_consensus.cc:388] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:28.106243 19711 raft_consensus.cc:482] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:28.106352 19711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:28.108278 19711 raft_consensus.cc:504] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b724003641a645388f53f4a3ebd13a8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260924 11:45:28.108664 19711 leader_election.cc:302] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b724003641a645388f53f4a3ebd13a8b; no voters:
I20260924 11:45:28.109125 19711 leader_election.cc:288] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:28.109216 19716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:28.109849 19716 raft_consensus.cc:686] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [term 1 LEADER]: Becoming Leader. State: Replica: b724003641a645388f53f4a3ebd13a8b, State: Running, Role: LEADER
I20260924 11:45:28.110303 19716 consensus_queue.cc:237] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b724003641a645388f53f4a3ebd13a8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260924 11:45:28.110924 19711 sys_catalog.cc:564] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:28.112830 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b724003641a645388f53f4a3ebd13a8b. Latest consensus state: current_term: 1 leader_uuid: "b724003641a645388f53f4a3ebd13a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b724003641a645388f53f4a3ebd13a8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } } }
I20260924 11:45:28.113380 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:28.112798 19717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b724003641a645388f53f4a3ebd13a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b724003641a645388f53f4a3ebd13a8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } } }
I20260924 11:45:28.114302 19717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:28.115864 19724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:28.120733 19724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:28.128544 19724 catalog_manager.cc:1347] Generated new cluster ID: a495fdbae53f4284ba55f02430602a5d
I20260924 11:45:28.128645 19724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:28.138154 19724 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:28.139683 19724 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:28.166723 19724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b: Generated new TSK 0
I20260924 11:45:28.167724 19724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 11:45:28.225380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:28.538780 19737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:28.539032 19737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:28.539120 19737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:28.546293 19737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:28.546406 19737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:28.546558 19737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:45:28.555394 19737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:28.556715 19737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:28.557965 19737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:28.570372 19742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:28.570320 19743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:28.570320 19745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:28.571033 19737 server_base.cc:1034] running on GCE node
I20260924 11:45:28.571641 19737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:28.572541 19737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:28.573729 19737 hybrid_clock.cc:648] HybridClock initialized: now 1790250328573677 us; error 50 us; skew 500 ppm
I20260924 11:45:28.574038 19737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:28 dist-test-slave-qhrh krb5kdc[19631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:28.576929 19737 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 11:45:28.578197 19737 webserver.cc:460] Webserver started at http://127.0.97.1:39203/ using document root <none> and password file <none>
I20260924 11:45:28.578830 19737 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:28.578969 19737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:28.579255 19737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:28.581992 19737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "5d3e625ac07f4e3692681c95ab793852"
format_stamp: "Formatted at 2026-09-24 11:45:28 on dist-test-slave-qhrh"
server_key: "c95a474a8e51a5cc15764a77cabfd6d1"
server_key_iv: "955f0a5e7bb8c3bb4bb25277c5f11b8b"
server_key_version: "encryptionkey@0"
I20260924 11:45:28.582751 19737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5d3e625ac07f4e3692681c95ab793852"
format_stamp: "Formatted at 2026-09-24 11:45:28 on dist-test-slave-qhrh"
server_key: "c95a474a8e51a5cc15764a77cabfd6d1"
server_key_iv: "955f0a5e7bb8c3bb4bb25277c5f11b8b"
server_key_version: "encryptionkey@0"
I20260924 11:45:28.587673 19737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:45:28.590839 19752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:28.591986 19737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:28.592211 19737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "5d3e625ac07f4e3692681c95ab793852"
format_stamp: "Formatted at 2026-09-24 11:45:28 on dist-test-slave-qhrh"
server_key: "c95a474a8e51a5cc15764a77cabfd6d1"
server_key_iv: "955f0a5e7bb8c3bb4bb25277c5f11b8b"
server_key_version: "encryptionkey@0"
I20260924 11:45:28.592432 19737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:28.630198 19737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:28.633991 19737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:28.634276 19737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:28.635185 19737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:28.636489 19737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:28.636588 19737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:28.636740 19737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:28.636806 19737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:28.668754 19737 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42357
I20260924 11:45:28.668841 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42357 every 8 connection(s)
I20260924 11:45:28.670101 19737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:45:28.677541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19737
I20260924 11:45:28.677762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:45:28.678144 388 external_mini_cluster.cc:1428] Setting key e3706d60a47b8fe63f5c605de095fcfb
Sep 24 11:45:28 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:28.690840 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:28.672595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34803 (local address 127.0.97.62:42293)
0924 11:45:28.673025 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:28.673030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:28.673900 (+ 870us) server_negotiation.cc:407] Connection header received
0924 11:45:28.675636 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:28.675642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:28.675723 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:28.675829 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:28.677248 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:28.678002 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:28.679628 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:28.680117 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:28.680273 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:28.684194 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:28.684202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:28.684210 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:28.684252 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:28.687502 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:28.688453 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:28.688461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:28.688467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:28.688622 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:28.689130 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:28.689136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:28.689140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:28.689493 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:28.689674 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:28.690411 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 11:45:28.690597 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260924 11:45:28.692258 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:28.673086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42293 (local address 127.0.97.1:34803)
0924 11:45:28.673743 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:28.673778 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:28.675364 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:28.676046 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:28.676053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:28.676570 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:28.677069 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:28.677086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:28.678150 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:28.678155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:28.679457 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:28.679468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:28.680374 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:28.680382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:28.680681 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:28.681403 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:28.681436 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:28.683990 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:28.687726 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:28.687736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:28.687764 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:28.688314 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:28.688733 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:28.688739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:28.688744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:28.688982 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:28.689654 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:28.689659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:28.690018 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 11:45:28.690315 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 11:45:28.690891 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":271,"threads_started":1}
I20260924 11:45:28.693795 19865 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42293
I20260924 11:45:28.694074 19865 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:28.694721 19865 heartbeater.cc:511] Master 127.0.97.62:42293 requested a full tablet report, sending...
I20260924 11:45:28.696913 19676 ts_manager.cc:194] Registered new tserver with Master: 5d3e625ac07f4e3692681c95ab793852 (127.0.97.1:42357)
I20260924 11:45:28.698194 19676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34803
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 11:45:28.731172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:29.039408 19871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:29.039640 19871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:29.039719 19871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:29.047257 19871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:29.047371 19871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:29.047534 19871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:45:29.058204 19871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:29.059448 19871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:29.060755 19871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:29.073274 19876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:29.073401 19877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:29.074263 19879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:29.074506 19871 server_base.cc:1034] running on GCE node
I20260924 11:45:29.075151 19871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:29.076165 19871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:29.077387 19871 hybrid_clock.cc:648] HybridClock initialized: now 1790250329077350 us; error 35 us; skew 500 ppm
I20260924 11:45:29.077732 19871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:29 dist-test-slave-qhrh krb5kdc[19631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:29.081238 19871 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 11:45:29.089301 19871 webserver.cc:460] Webserver started at http://127.0.97.2:46409/ using document root <none> and password file <none>
I20260924 11:45:29.089993 19871 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:29.090129 19871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:29.090441 19871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:29.093355 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "a3e651d00b3246b792c5b302516246e6"
format_stamp: "Formatted at 2026-09-24 11:45:29 on dist-test-slave-qhrh"
server_key: "1d6069d024e025f8b33b608b4f3e3b33"
server_key_iv: "03dbbc0f220214677103256e1cd15851"
server_key_version: "encryptionkey@0"
I20260924 11:45:29.094063 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a3e651d00b3246b792c5b302516246e6"
format_stamp: "Formatted at 2026-09-24 11:45:29 on dist-test-slave-qhrh"
server_key: "1d6069d024e025f8b33b608b4f3e3b33"
server_key_iv: "03dbbc0f220214677103256e1cd15851"
server_key_version: "encryptionkey@0"
I20260924 11:45:29.099014 19871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:45:29.102381 19886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:29.103664 19871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:45:29.103782 19871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "a3e651d00b3246b792c5b302516246e6"
format_stamp: "Formatted at 2026-09-24 11:45:29 on dist-test-slave-qhrh"
server_key: "1d6069d024e025f8b33b608b4f3e3b33"
server_key_iv: "03dbbc0f220214677103256e1cd15851"
server_key_version: "encryptionkey@0"
I20260924 11:45:29.103940 19871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:29.121955 19871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:29.126262 19871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:29.126499 19871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:29.127321 19871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:29.128630 19871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:29.128703 19871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:29.128783 19871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:29.128825 19871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:29.160646 19871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34639
I20260924 11:45:29.160673 19998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34639 every 8 connection(s)
I20260924 11:45:29.162074 19871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:45:29.172202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 19871
I20260924 11:45:29.172361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:45:29.172894 388 external_mini_cluster.cc:1428] Setting key 374a43fa0eca0fd299114aa165141119
Sep 24 11:45:29 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:29.186625 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.164656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45697 (local address 127.0.97.62:42293)
0924 11:45:29.164927 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.164937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.166521 (+ 1584us) server_negotiation.cc:407] Connection header received
0924 11:45:29.169427 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.169431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.169495 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.169576 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:29.171697 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.172807 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.175461 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.176249 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.176388 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.180102 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:29.180111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:29.180119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:29.180190 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:29.183530 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.184465 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.184468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.184470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.184588 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.185040 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.185042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.185044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.185279 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:29.185375 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.186199 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.186359 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:45:29.188030 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.165335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42293 (local address 127.0.97.2:45697)
0924 11:45:29.166335 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.166391 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.169077 (+ 2686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.169868 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.169882 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.170628 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:29.171476 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.171505 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.172992 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.173001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.175270 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.175290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.176538 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.176547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.176800 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.177554 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:29.177576 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:29.179865 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:29.183687 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:29.183695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:29.183713 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:29.184351 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:29.184694 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:29.184701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:29.184706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:29.184943 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:29.185401 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:29.185418 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:29.185822 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.186138 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.186691 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":383,"threads_started":1}
I20260924 11:45:29.189386 19999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42293
I20260924 11:45:29.189695 19999 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:29.190361 19999 heartbeater.cc:511] Master 127.0.97.62:42293 requested a full tablet report, sending...
I20260924 11:45:29.191831 19676 ts_manager.cc:194] Registered new tserver with Master: a3e651d00b3246b792c5b302516246e6 (127.0.97.2:34639)
I20260924 11:45:29.193174 19676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45697
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 11:45:29.227486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:29.537099 20004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:29.537356 20004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:29.537528 20004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:29.544735 20004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:29.544842 20004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:29.545008 20004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:45:29.553929 20004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:29.555094 20004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:29.556346 20004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:29.569519 20012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:29.569499 20010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:29.570005 20004 server_base.cc:1034] running on GCE node
W20260924 11:45:29.570258 20009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:29.570993 20004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:29.571869 20004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:29.573067 20004 hybrid_clock.cc:648] HybridClock initialized: now 1790250329573029 us; error 31 us; skew 500 ppm
I20260924 11:45:29.573369 20004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:29 dist-test-slave-qhrh krb5kdc[19631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:29.576386 20004 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 11:45:29.577749 20004 webserver.cc:460] Webserver started at http://127.0.97.3:46281/ using document root <none> and password file <none>
I20260924 11:45:29.578320 20004 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:29.578405 20004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:29.578634 20004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:29.581327 20004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "55a40a3b082440ee987665f52c08f9d9"
format_stamp: "Formatted at 2026-09-24 11:45:29 on dist-test-slave-qhrh"
server_key: "4174bbfba20e567d34e51633e83b2838"
server_key_iv: "a938b2c6811c5d5fe13acaa6343c13ce"
server_key_version: "encryptionkey@0"
I20260924 11:45:29.582011 20004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "55a40a3b082440ee987665f52c08f9d9"
format_stamp: "Formatted at 2026-09-24 11:45:29 on dist-test-slave-qhrh"
server_key: "4174bbfba20e567d34e51633e83b2838"
server_key_iv: "a938b2c6811c5d5fe13acaa6343c13ce"
server_key_version: "encryptionkey@0"
I20260924 11:45:29.586684 20004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:45:29.589715 20019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:29.590973 20004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:29.591085 20004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "55a40a3b082440ee987665f52c08f9d9"
format_stamp: "Formatted at 2026-09-24 11:45:29 on dist-test-slave-qhrh"
server_key: "4174bbfba20e567d34e51633e83b2838"
server_key_iv: "a938b2c6811c5d5fe13acaa6343c13ce"
server_key_version: "encryptionkey@0"
I20260924 11:45:29.591228 20004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:29.619017 20004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:29.622840 20004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:29.623071 20004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:29.623894 20004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:29.625061 20004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:29.625126 20004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:29.625198 20004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:29.625239 20004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:29.655821 20004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36127
I20260924 11:45:29.655877 20131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36127 every 8 connection(s)
I20260924 11:45:29.656972 20004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:45:29.661821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20004
I20260924 11:45:29.661996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:45:29.662490 388 external_mini_cluster.cc:1428] Setting key 6b5e91d188247c571ecf3c19c2110212
Sep 24 11:45:29 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:29.675426 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.659302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57427 (local address 127.0.97.62:42293)
0924 11:45:29.659483 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.659488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.660540 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 11:45:29.662381 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.662387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.662478 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.662621 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:29.664327 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.665155 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.666459 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.666799 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.666877 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.670399 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:29.670404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:29.670407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:29.670444 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:29.673058 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.673666 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.673670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.673672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.673805 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.674146 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.674149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.674151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.674369 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:29.674474 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.675031 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.675174 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:45:29.676375 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.659712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42293 (local address 127.0.97.3:57427)
0924 11:45:29.660374 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.660408 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.662114 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.662822 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.662829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.663441 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:29.664164 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.664211 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.665278 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.665281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.666352 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.666363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.666951 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.666955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.667127 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.667734 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:45:29.667752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:29.670209 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:29.673174 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:29.673180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:29.673189 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:29.673561 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:29.673885 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:45:29.673887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:45:29.673889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:45:29.674043 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:45:29.674459 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:29.674463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:29.674725 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.674916 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.675251 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":237,"threads_started":1}
I20260924 11:45:29.677632 20132 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42293
I20260924 11:45:29.677910 20132 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:29.678510 20132 heartbeater.cc:511] Master 127.0.97.62:42293 requested a full tablet report, sending...
I20260924 11:45:29.679677 19675 ts_manager.cc:194] Registered new tserver with Master: 55a40a3b082440ee987665f52c08f9d9 (127.0.97.3:36127)
I20260924 11:45:29.680528 19675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57427
I20260924 11:45:29.689210 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:45:29.700861 19865 heartbeater.cc:503] Master 127.0.97.62:42293 was elected leader, sending a full tablet report...
I20260924 11:45:29.705004 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.692952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.97.62:42293)
0924 11:45:29.693094 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.693098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.693527 (+ 429us) server_negotiation.cc:407] Connection header received
0924 11:45:29.693691 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.693712 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.693762 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.693828 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:29.695213 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.695914 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.697213 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.697558 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.697640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.699587 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:29.699591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:29.699593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:29.699618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:29.702474 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.703145 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.703152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.703155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.703273 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.703624 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.703631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.703634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.703930 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:29.704206 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.704549 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.704741 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:45:29.708230 19675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:45:29.714743 19675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:45:29.739372 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.725756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34639 (local address 127.0.0.1:36922)
0924 11:45:29.727545 (+ 1789us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.727590 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.727766 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.729381 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.729388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.729422 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:29.729959 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.729976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.732655 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.732663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.735545 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.735563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.737976 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.737985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.738360 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.738447 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.738597 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.738847 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"thread_start_us":2885,"threads_started":1}
I20260924 11:45:29.741281 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.726006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.0.97.2:34639)
0924 11:45:29.728710 (+ 2704us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.728716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.728738 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:45:29.728876 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.728882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.729149 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.730195 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:29.730462 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.732444 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.735801 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.737605 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.740310 (+ 2705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.740565 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.740647 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.740878 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2574,"thread_start_us":2483,"threads_started":1}
I20260924 11:45:29.743134 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.726990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.0.97.3:36127)
0924 11:45:29.728471 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.728482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.728988 (+ 506us) server_negotiation.cc:407] Connection header received
0924 11:45:29.730382 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.730394 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.730642 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.730824 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:29.731893 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.734000 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.738196 (+ 4196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.740188 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.740315 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.740597 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.742595 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.742774 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":1085,"threads_started":1}
I20260924 11:45:29.743527 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.726341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.0.1:34112)
0924 11:45:29.728873 (+ 2532us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.728891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.730159 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.731181 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.731188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.731220 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:29.731670 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.731684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.735234 (+ 3550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.735241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.737929 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.737975 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.742034 (+ 4059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.742043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.742313 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.742388 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.743055 (+ 667us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.743200 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2396,"thread_start_us":930,"threads_started":1}
I20260924 11:45:29.746508 20067 tablet_service.cc:1467] Processing CreateTablet for tablet 3191a638caf848beb17afdb723233c8e (DEFAULT_TABLE table=test-table [id=71a33801befe4f1cb3a0c811fdaa025f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:29.746534 19934 tablet_service.cc:1467] Processing CreateTablet for tablet 3191a638caf848beb17afdb723233c8e (DEFAULT_TABLE table=test-table [id=71a33801befe4f1cb3a0c811fdaa025f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:29.747848 20067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3191a638caf848beb17afdb723233c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:29.747902 19934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3191a638caf848beb17afdb723233c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:29.748411 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.729106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42357 (local address 127.0.0.1:39744)
0924 11:45:29.739662 (+ 10556us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.739684 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.739800 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.742150 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.742154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.742177 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:29.742338 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.742349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.744265 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.744269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.745893 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.745906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.747702 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.747707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.747886 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.747924 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.748006 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.748121 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10417,"thread_start_us":13557,"threads_started":1}
I20260924 11:45:29.749115 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.729333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.97.1:42357)
0924 11:45:29.741568 (+ 12235us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.741577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.741606 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:45:29.741704 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.741709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.741934 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.742164 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:29.742501 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.744129 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.746073 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.747467 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.747666 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.747863 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.748619 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.748783 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12042,"thread_start_us":13012,"threads_started":1}
I20260924 11:45:29.752326 19800 tablet_service.cc:1467] Processing CreateTablet for tablet 3191a638caf848beb17afdb723233c8e (DEFAULT_TABLE table=test-table [id=71a33801befe4f1cb3a0c811fdaa025f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:29.753636 19800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3191a638caf848beb17afdb723233c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:29.758402 20150 tablet_bootstrap.cc:492] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Bootstrap starting.
I20260924 11:45:29.761765 20150 tablet_bootstrap.cc:654] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:29.762804 20150 log.cc:824] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:29.765533 20150 tablet_bootstrap.cc:492] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: No bootstrap required, opened a new log
I20260924 11:45:29.765766 20151 tablet_bootstrap.cc:492] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Bootstrap starting.
I20260924 11:45:29.765811 20150 ts_tablet_manager.cc:1397] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 11:45:29.767256 20153 tablet_bootstrap.cc:492] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Bootstrap starting.
I20260924 11:45:29.769347 20151 tablet_bootstrap.cc:654] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:29.770256 20151 log.cc:824] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:29.770439 20153 tablet_bootstrap.cc:654] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:29.771507 20153 log.cc:824] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:29.772962 20151 tablet_bootstrap.cc:492] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: No bootstrap required, opened a new log
I20260924 11:45:29.773202 20151 ts_tablet_manager.cc:1397] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 11:45:29.774260 20153 tablet_bootstrap.cc:492] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: No bootstrap required, opened a new log
I20260924 11:45:29.774492 20153 ts_tablet_manager.cc:1397] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 11:45:29.775489 20150 raft_consensus.cc:348] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.775894 20150 raft_consensus.cc:374] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:29.775985 20150 raft_consensus.cc:729] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a40a3b082440ee987665f52c08f9d9, State: Initialized, Role: FOLLOWER
I20260924 11:45:29.776646 20150 consensus_queue.cc:260] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.777709 20132 heartbeater.cc:503] Master 127.0.97.62:42293 was elected leader, sending a full tablet report...
I20260924 11:45:29.778264 20150 ts_tablet_manager.cc:1428] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 11:45:29.782992 20151 raft_consensus.cc:348] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.783401 20151 raft_consensus.cc:374] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:29.783510 20151 raft_consensus.cc:729] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e651d00b3246b792c5b302516246e6, State: Initialized, Role: FOLLOWER
I20260924 11:45:29.784058 20151 consensus_queue.cc:260] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.784407 20153 raft_consensus.cc:348] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.784780 20153 raft_consensus.cc:374] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:29.784878 20153 raft_consensus.cc:729] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3e625ac07f4e3692681c95ab793852, State: Initialized, Role: FOLLOWER
I20260924 11:45:29.785082 19999 heartbeater.cc:503] Master 127.0.97.62:42293 was elected leader, sending a full tablet report...
I20260924 11:45:29.785463 20153 consensus_queue.cc:260] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.785739 20151 ts_tablet_manager.cc:1428] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260924 11:45:29.786998 20153 ts_tablet_manager.cc:1428] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 11:45:29.879968 20157 raft_consensus.cc:482] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:29.880354 20157 raft_consensus.cc:504] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.882270 20157 leader_election.cc:288] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3e625ac07f4e3692681c95ab793852 (127.0.97.1:42357), 55a40a3b082440ee987665f52c08f9d9 (127.0.97.3:36127)
I20260924 11:45:29.888854 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.882870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50441 (local address 127.0.97.3:36127)
0924 11:45:29.883079 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.883085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.883994 (+ 909us) server_negotiation.cc:407] Connection header received
0924 11:45:29.884209 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.884214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.884274 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.884358 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:29.884845 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.885567 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.887290 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.887989 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.888126 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.888238 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.888510 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.888623 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:45:29.888883 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.882800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46943 (local address 127.0.97.1:42357)
0924 11:45:29.883012 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.883019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.884045 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 11:45:29.884257 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.884262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.884323 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.884403 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:29.885018 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.885702 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.887308 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.888022 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.888187 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.888286 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.888547 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.888664 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:45:29.889000 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.882787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.97.2:50441)
0924 11:45:29.883895 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.883912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.884060 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.884371 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.884374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.884393 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:29.884701 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.884712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.885688 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.885695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.887156 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.887167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.888214 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.888222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.888363 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.888384 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.888486 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.888595 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":65,"thread_start_us":480,"threads_started":1}
I20260924 11:45:29.889210 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.882907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42357 (local address 127.0.97.2:46943)
0924 11:45:29.883945 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:29.883959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:29.884089 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:29.884414 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:29.884418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:29.884436 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:29.884861 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.884870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.885836 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.885842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.887166 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:29.887176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.888246 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.888251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:29.888403 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.888423 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:45:29.888514 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 11:45:29.888618 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":2,"thread_start_us":525,"threads_started":1}
I20260924 11:45:29.889894 20087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191a638caf848beb17afdb723233c8e" candidate_uuid: "a3e651d00b3246b792c5b302516246e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a40a3b082440ee987665f52c08f9d9" is_pre_election: true
I20260924 11:45:29.890067 19820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191a638caf848beb17afdb723233c8e" candidate_uuid: "a3e651d00b3246b792c5b302516246e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3e625ac07f4e3692681c95ab793852" is_pre_election: true
I20260924 11:45:29.890421 20087 raft_consensus.cc:2454] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e651d00b3246b792c5b302516246e6 in term 0.
I20260924 11:45:29.890605 19820 raft_consensus.cc:2454] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e651d00b3246b792c5b302516246e6 in term 0.
I20260924 11:45:29.891034 19887 leader_election.cc:302] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a40a3b082440ee987665f52c08f9d9, a3e651d00b3246b792c5b302516246e6; no voters:
I20260924 11:45:29.891412 20157 raft_consensus.cc:2781] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:45:29.891544 20157 raft_consensus.cc:482] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:45:29.891652 20157 raft_consensus.cc:3037] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:29.893821 20157 raft_consensus.cc:504] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.894400 20157 leader_election.cc:288] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [CANDIDATE]: Term 1 election: Requested vote from peers 5d3e625ac07f4e3692681c95ab793852 (127.0.97.1:42357), 55a40a3b082440ee987665f52c08f9d9 (127.0.97.3:36127)
I20260924 11:45:29.894793 19820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191a638caf848beb17afdb723233c8e" candidate_uuid: "a3e651d00b3246b792c5b302516246e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3e625ac07f4e3692681c95ab793852"
I20260924 11:45:29.894914 20087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191a638caf848beb17afdb723233c8e" candidate_uuid: "a3e651d00b3246b792c5b302516246e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a40a3b082440ee987665f52c08f9d9"
I20260924 11:45:29.894974 19820 raft_consensus.cc:3037] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:29.895131 20087 raft_consensus.cc:3037] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:29.897071 19820 raft_consensus.cc:2454] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e651d00b3246b792c5b302516246e6 in term 1.
I20260924 11:45:29.897400 19889 leader_election.cc:302] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3e625ac07f4e3692681c95ab793852, a3e651d00b3246b792c5b302516246e6; no voters:
I20260924 11:45:29.897524 20087 raft_consensus.cc:2454] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e651d00b3246b792c5b302516246e6 in term 1.
I20260924 11:45:29.897711 20157 raft_consensus.cc:2781] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:29.898176 20157 raft_consensus.cc:686] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 1 LEADER]: Becoming Leader. State: Replica: a3e651d00b3246b792c5b302516246e6, State: Running, Role: LEADER
I20260924 11:45:29.898653 20157 consensus_queue.cc:237] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260924 11:45:29.902968 19675 catalog_manager.cc:5495] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e651d00b3246b792c5b302516246e6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a3e651d00b3246b792c5b302516246e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e651d00b3246b792c5b302516246e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: UNKNOWN } } }
W20260924 11:45:29.908371 20133 tablet.cc:2369] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:45:29.914026 20000 tablet.cc:2369] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:45:29.922327 19866 tablet.cc:2369] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:29.929122 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.923831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.0.97.2:34639)
0924 11:45:29.924031 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.924036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.924054 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:45:29.924312 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.924316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.924388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.924529 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:45:29.924870 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.925535 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.926956 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.927355 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.927482 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.928073 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:45:29.928576 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:45:29.928729 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.928780 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.928854 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:45:29.936785 20087 raft_consensus.cc:1261] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 1 FOLLOWER]: Refusing update from remote peer a3e651d00b3246b792c5b302516246e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:29.936786 19820 raft_consensus.cc:1261] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 1 FOLLOWER]: Refusing update from remote peer a3e651d00b3246b792c5b302516246e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:29.937795 20161 consensus_queue.cc:1035] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a40a3b082440ee987665f52c08f9d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:29.938230 20157 consensus_queue.cc:1035] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3e625ac07f4e3692681c95ab793852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:29.950369 20165 mvcc.cc:205] Tried to move back new op lower bound from 7332865351410618368 to 7332865351268818944. Current Snapshot: MvccSnapshot[applied={T|T < 7332865351410618368}]
I20260924 11:45:29.961809 19676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:29.962067 19676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:29.965546 19676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b724003641a645388f53f4a3ebd13a8b: Sending DeleteTablet for 3 replicas of tablet 3191a638caf848beb17afdb723233c8e
I20260924 11:45:29.966645 19934 tablet_service.cc:1514] Processing DeleteTablet for tablet 3191a638caf848beb17afdb723233c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36922
I20260924 11:45:29.966741 19800 tablet_service.cc:1514] Processing DeleteTablet for tablet 3191a638caf848beb17afdb723233c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39744
I20260924 11:45:29.967345 20067 tablet_service.cc:1514] Processing DeleteTablet for tablet 3191a638caf848beb17afdb723233c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34112
I20260924 11:45:29.967533 20175 tablet_replica.cc:330] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: stopping tablet replica
I20260924 11:45:29.967892 20175 raft_consensus.cc:2229] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:29.968377 20175 raft_consensus.cc:2258] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:29.969170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19737
I20260924 11:45:29.970336 20175 ts_tablet_manager.cc:1905] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:29.977170 20175 ts_tablet_manager.cc:1918] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:29.977399 20175 log.cc:1196] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-0/wal/wals/3191a638caf848beb17afdb723233c8e
I20260924 11:45:29.977820 20174 tablet_replica.cc:330] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: stopping tablet replica
I20260924 11:45:29.978016 20175 ts_tablet_manager.cc:1939] T 3191a638caf848beb17afdb723233c8e P 5d3e625ac07f4e3692681c95ab793852: Deleting consensus metadata
I20260924 11:45:29.978237 20176 tablet_replica.cc:330] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: stopping tablet replica
I20260924 11:45:29.978401 20174 raft_consensus.cc:2229] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:45:29.978892 20176 raft_consensus.cc:2229] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:29.979126 20174 raft_consensus.cc:2258] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:29.979970 19662 catalog_manager.cc:4841] TS 5d3e625ac07f4e3692681c95ab793852 (127.0.97.1:42357): tablet 3191a638caf848beb17afdb723233c8e (table test-table [id=71a33801befe4f1cb3a0c811fdaa025f]) successfully deleted
I20260924 11:45:29.980418 20176 raft_consensus.cc:2258] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:29.981144 20174 ts_tablet_manager.cc:1905] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:29.982969 20176 ts_tablet_manager.cc:1905] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:29.988286 20174 ts_tablet_manager.cc:1918] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:29.988593 20174 log.cc:1196] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/3191a638caf848beb17afdb723233c8e
I20260924 11:45:29.989095 20174 ts_tablet_manager.cc:1939] T 3191a638caf848beb17afdb723233c8e P a3e651d00b3246b792c5b302516246e6: Deleting consensus metadata
I20260924 11:45:29.989518 20176 ts_tablet_manager.cc:1918] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:29.989691 20176 log.cc:1196] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/3191a638caf848beb17afdb723233c8e
I20260924 11:45:29.990182 20176 ts_tablet_manager.cc:1939] T 3191a638caf848beb17afdb723233c8e P 55a40a3b082440ee987665f52c08f9d9: Deleting consensus metadata
I20260924 11:45:29.990916 19662 catalog_manager.cc:4841] TS a3e651d00b3246b792c5b302516246e6 (127.0.97.2:34639): tablet 3191a638caf848beb17afdb723233c8e (table test-table [id=71a33801befe4f1cb3a0c811fdaa025f]) successfully deleted
I20260924 11:45:29.991829 19660 catalog_manager.cc:4841] TS 55a40a3b082440ee987665f52c08f9d9 (127.0.97.3:36127): tablet 3191a638caf848beb17afdb723233c8e (table test-table [id=71a33801befe4f1cb3a0c811fdaa025f]) successfully deleted
Sep 24 11:45:29 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 11:45:29.998805 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:29.970752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.0.97.1:42357)
0924 11:45:29.970957 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:29.970966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:29.982204 (+ 11238us) server_negotiation.cc:407] Connection header received
0924 11:45:29.982402 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:29.982409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:29.982504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:29.982620 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:29.984152 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.985215 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.987093 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:29.987726 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:29.987836 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:29.993160 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:29.993171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:29.993184 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:29.993214 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:29.996298 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.997010 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.997019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.997023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.997147 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:29.997528 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:29.997531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:29.997533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:29.997836 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:29.998039 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:29.998338 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 11:45:29.998478 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:45:29.999404 19860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:30.095497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19737
I20260924 11:45:30.113590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19871
Sep 24 11:45:30 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 11:45:30.128784 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:30.114340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.0.97.2:34639)
0924 11:45:30.114529 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:30.114534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:30.114553 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:45:30.114750 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:30.114755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:30.114822 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:30.114928 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:30.116178 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:30.116941 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:30.118179 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:30.118588 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:30.118716 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:30.122860 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:30.122872 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:30.122890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:30.122927 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:30.125957 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:30.126641 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:30.126651 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:30.126659 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:30.126794 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:30.127183 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:30.127187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:30.127190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:30.127534 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:30.127881 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:30.128122 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 11:45:30.128498 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:45:30.129469 19994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:30.239264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19871
I20260924 11:45:30.256843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20004
Sep 24 11:45:30 dist-test-slave-qhrh krb5kdc[19631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 11:45:30.270915 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:30.257531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.0.97.3:36127)
0924 11:45:30.257709 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:30.257715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:30.257801 (+ 86us) server_negotiation.cc:407] Connection header received
0924 11:45:30.258012 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:30.258016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:30.258077 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:30.258176 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:30.259417 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:30.260086 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:30.261384 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:30.261783 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:30.261909 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:30.266135 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:30.266158 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:30.266173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:30.266205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:30.268508 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:30.269098 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:30.269104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:30.269106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:30.269225 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:30.269561 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:30.269566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:30.269569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:30.269880 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:30.270151 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:30.270431 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 11:45:30.270629 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:30.271469 20127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:30.355938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20004
I20260924 11:45:30.372191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 19644
I20260924 11:45:30.373265 19706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:30.475517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 19644
2026-09-24T11:45:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T11:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:30Z Disabled control of system clock
I20260924 11:45:30.546578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37253
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:45:30.858489 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:30.858731 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:30.858804 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:30.865705 20185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:30.865789 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:30.865832 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:30.865885 20185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:30.865928 20185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:30.874310 20185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:30.875310 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:30.876850 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:30.887668 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:30.887735 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:30.887714 20190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:30.888273 20185 server_base.cc:1034] running on GCE node
I20260924 11:45:30.889047 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:30.890229 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:30.891414 20185 hybrid_clock.cc:648] HybridClock initialized: now 1790250330891378 us; error 31 us; skew 500 ppm
I20260924 11:45:30.891739 20185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:30.893778 20185 webserver.cc:460] Webserver started at http://127.0.97.62:44157/ using document root <none> and password file <none>
I20260924 11:45:30.894376 20185 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:30.894450 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:30.894799 20185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:30.897449 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "3f6bf48d023f452488df1fd2a6d9d1d0"
format_stamp: "Formatted at 2026-09-24 11:45:30 on dist-test-slave-qhrh"
server_key: "6469af8dc70facf7e2124fee2a285b70"
server_key_iv: "f9b1bc0f83e043a060426d33bd9074c4"
server_key_version: "encryptionkey@0"
I20260924 11:45:30.898177 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "3f6bf48d023f452488df1fd2a6d9d1d0"
format_stamp: "Formatted at 2026-09-24 11:45:30 on dist-test-slave-qhrh"
server_key: "6469af8dc70facf7e2124fee2a285b70"
server_key_iv: "f9b1bc0f83e043a060426d33bd9074c4"
server_key_version: "encryptionkey@0"
I20260924 11:45:30.902803 20185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 11:45:30.905915 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:30.907059 20185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:30.907227 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "3f6bf48d023f452488df1fd2a6d9d1d0"
format_stamp: "Formatted at 2026-09-24 11:45:30 on dist-test-slave-qhrh"
server_key: "6469af8dc70facf7e2124fee2a285b70"
server_key_iv: "f9b1bc0f83e043a060426d33bd9074c4"
server_key_version: "encryptionkey@0"
I20260924 11:45:30.907410 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:30.934506 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:30.935235 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:30.935530 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:30.953038 20185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37253
I20260924 11:45:30.953058 20250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37253 every 8 connection(s)
I20260924 11:45:30.954298 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:30.955694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20185
I20260924 11:45:30.955871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:30.956398 388 external_mini_cluster.cc:1428] Setting key 4e4385a7ed2586ddc83865c40002715a
I20260924 11:45:30.958701 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:30.968914 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:30.958143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.0.97.62:37253)
0924 11:45:30.959055 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:30.959064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:30.959100 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:45:30.960631 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:30.960650 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:30.961070 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:30.961482 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:30.962038 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:30.963313 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:30.964764 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:30.965379 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:30.965560 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:30.966682 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:30.966689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:30.966700 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:30.967248 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:30.967321 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:30.967341 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:30.967491 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:30.967721 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 11:45:30.967975 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":304,"threads_started":1}
I20260924 11:45:30.969135 20251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0: Bootstrap starting.
I20260924 11:45:30.972608 20251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:30.973549 20251 log.cc:824] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:30.975847 20251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0: No bootstrap required, opened a new log
I20260924 11:45:30.982503 20251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37253 } }
I20260924 11:45:30.982856 20251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:30.982955 20251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6bf48d023f452488df1fd2a6d9d1d0, State: Initialized, Role: FOLLOWER
I20260924 11:45:30.983455 20251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37253 } }
I20260924 11:45:30.983685 20251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:30.983832 20251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:30.984006 20251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:30.986171 20251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37253 } }
I20260924 11:45:30.986641 20251 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6bf48d023f452488df1fd2a6d9d1d0; no voters:
I20260924 11:45:30.987110 20251 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:30.987221 20256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:30.987968 20256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f6bf48d023f452488df1fd2a6d9d1d0, State: Running, Role: LEADER
I20260924 11:45:30.988579 20256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37253 } }
I20260924 11:45:30.989141 20251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:30.991294 20258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6bf48d023f452488df1fd2a6d9d1d0. Latest consensus state: current_term: 1 leader_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37253 } } }
I20260924 11:45:30.991641 20258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:30.992570 20257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6bf48d023f452488df1fd2a6d9d1d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37253 } } }
I20260924 11:45:30.992947 20257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:30.994150 20262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:30.998467 20262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:31.007102 20262 catalog_manager.cc:1347] Generated new cluster ID: 581420ee9dac451d88ff75262f17fe70
I20260924 11:45:31.007226 20262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:31.025425 20262 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:31.027096 20262 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:31.040302 20262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0: Generated new TSK 0
I20260924 11:45:31.041718 20262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:31.054019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37253
--builtin_ntp_servers=127.0.97.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:45:31.369455 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:31.369727 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:31.369869 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:31.376520 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:31.376636 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:31.376803 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 11:45:31.384994 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:31.386108 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:31.387708 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:31.400628 20280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:31.400661 20281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:31.400774 20283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:31.401196 20275 server_base.cc:1034] running on GCE node
I20260924 11:45:31.401907 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:31.402810 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:31.404003 20275 hybrid_clock.cc:648] HybridClock initialized: now 1790250331403970 us; error 35 us; skew 500 ppm
I20260924 11:45:31.404335 20275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:31.406426 20275 webserver.cc:460] Webserver started at http://127.0.97.1:34997/ using document root <none> and password file <none>
I20260924 11:45:31.407060 20275 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:31.407163 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:31.407459 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:31.410315 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data/instance:
uuid: "f68a0674765b4318be98a6728543c346"
format_stamp: "Formatted at 2026-09-24 11:45:31 on dist-test-slave-qhrh"
server_key: "88c871657af336325f7a38a2c6c6ffad"
server_key_iv: "109344de155955446f42fa2fdb8196d1"
server_key_version: "encryptionkey@0"
I20260924 11:45:31.411105 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f68a0674765b4318be98a6728543c346"
format_stamp: "Formatted at 2026-09-24 11:45:31 on dist-test-slave-qhrh"
server_key: "88c871657af336325f7a38a2c6c6ffad"
server_key_iv: "109344de155955446f42fa2fdb8196d1"
server_key_version: "encryptionkey@0"
I20260924 11:45:31.415843 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 11:45:31.418956 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:31.420120 20275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:45:31.420342 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal
uuid: "f68a0674765b4318be98a6728543c346"
format_stamp: "Formatted at 2026-09-24 11:45:31 on dist-test-slave-qhrh"
server_key: "88c871657af336325f7a38a2c6c6ffad"
server_key_iv: "109344de155955446f42fa2fdb8196d1"
server_key_version: "encryptionkey@0"
I20260924 11:45:31.420542 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:31.435220 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:31.435956 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:31.436291 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:31.437168 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:31.438490 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:31.438606 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:31.438730 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:31.438819 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:31.469898 20275 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32975
I20260924 11:45:31.469976 20401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32975 every 8 connection(s)
I20260924 11:45:31.471196 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/data/info.pb
I20260924 11:45:31.479645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20275
I20260924 11:45:31.479808 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-0/wal/instance
I20260924 11:45:31.480381 388 external_mini_cluster.cc:1428] Setting key a2e25b4f50d91c1875501288ececd587
I20260924 11:45:31.483390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37253
--builtin_ntp_servers=127.0.97.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:45:31.484498 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:31.473814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52861 (local address 127.0.97.62:37253)
0924 11:45:31.474257 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:31.474262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:31.475665 (+ 1403us) server_negotiation.cc:407] Connection header received
0924 11:45:31.477429 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:31.477433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:31.477494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:31.477602 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:31.478536 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.479222 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.481361 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.481952 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.482105 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:31.483711 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:31.483716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:31.483720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:31.483736 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:31.483772 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:31.483783 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:31.483853 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:31.484215 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 11:45:31.484280 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260924 11:45:31.485868 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:31.474414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37253 (local address 127.0.97.1:52861)
0924 11:45:31.475482 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:31.475532 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:31.477201 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:31.477800 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:31.477807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:31.477883 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:31.478394 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:31.478407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.479370 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.479374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:31.481198 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:31.481210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.482178 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.482184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:31.482488 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:31.483269 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:31.483274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:31.483331 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:31.483434 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:31.483921 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:31.483954 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:31.483965 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:45:31.484189 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 11:45:31.484584 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":352,"threads_started":1}
I20260924 11:45:31.487370 20402 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37253
I20260924 11:45:31.487640 20402 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:31.488461 20402 heartbeater.cc:511] Master 127.0.97.62:37253 requested a full tablet report, sending...
I20260924 11:45:31.490700 20216 ts_manager.cc:194] Registered new tserver with Master: f68a0674765b4318be98a6728543c346 (127.0.97.1:32975)
I20260924 11:45:31.492666 20216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52861
W20260924 11:45:31.787609 20406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:31.787851 20406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:31.787931 20406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:31.794669 20406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:31.794750 20406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:31.794904 20406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 11:45:31.803241 20406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:31.804513 20406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:31.806183 20406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:31.818219 20414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:31.818261 20411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:31.818259 20412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:31.819077 20406 server_base.cc:1034] running on GCE node
I20260924 11:45:31.819623 20406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:31.820510 20406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:31.821712 20406 hybrid_clock.cc:648] HybridClock initialized: now 1790250331821673 us; error 36 us; skew 500 ppm
I20260924 11:45:31.822005 20406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:31.824211 20406 webserver.cc:460] Webserver started at http://127.0.97.2:34091/ using document root <none> and password file <none>
I20260924 11:45:31.824852 20406 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:31.824970 20406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:31.825280 20406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:31.827939 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data/instance:
uuid: "f604a589343b4d3b97f2a5f574ed5b46"
format_stamp: "Formatted at 2026-09-24 11:45:31 on dist-test-slave-qhrh"
server_key: "a2df81a1fee94a8ab08b886921244785"
server_key_iv: "6178b0d2edd354de34d275cb34289bf1"
server_key_version: "encryptionkey@0"
I20260924 11:45:31.828760 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f604a589343b4d3b97f2a5f574ed5b46"
format_stamp: "Formatted at 2026-09-24 11:45:31 on dist-test-slave-qhrh"
server_key: "a2df81a1fee94a8ab08b886921244785"
server_key_iv: "6178b0d2edd354de34d275cb34289bf1"
server_key_version: "encryptionkey@0"
I20260924 11:45:31.833470 20406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:45:31.836598 20420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:31.837718 20406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 11:45:31.837872 20406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal
uuid: "f604a589343b4d3b97f2a5f574ed5b46"
format_stamp: "Formatted at 2026-09-24 11:45:31 on dist-test-slave-qhrh"
server_key: "a2df81a1fee94a8ab08b886921244785"
server_key_iv: "6178b0d2edd354de34d275cb34289bf1"
server_key_version: "encryptionkey@0"
I20260924 11:45:31.838052 20406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:31.852336 20406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:31.853082 20406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:31.853345 20406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:31.854235 20406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:31.855605 20406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:31.855734 20406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:31.855857 20406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:31.855940 20406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:31.886591 20406 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46151
I20260924 11:45:31.886644 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46151 every 8 connection(s)
I20260924 11:45:31.887879 20406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/data/info.pb
I20260924 11:45:31.888274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20406
I20260924 11:45:31.888465 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal/instance
I20260924 11:45:31.888988 388 external_mini_cluster.cc:1428] Setting key 88f5ab8bd4c360a09aa1a2430b0e6daf
I20260924 11:45:31.891990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37253
--builtin_ntp_servers=127.0.97.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:45:31.903061 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:31.891189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46905 (local address 127.0.97.62:37253)
0924 11:45:31.891369 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:31.891375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:31.892955 (+ 1580us) server_negotiation.cc:407] Connection header received
0924 11:45:31.895566 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:31.895570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:31.895629 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:31.895696 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:31.897139 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.897823 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.900229 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.900632 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.900722 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:31.902239 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:31.902244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:31.902247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:31.902264 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:31.902296 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:31.902307 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:31.902413 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:31.902642 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 11:45:31.902734 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:45:31.903976 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:31.891720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37253 (local address 127.0.97.2:46905)
0924 11:45:31.892798 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:31.892844 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:31.895300 (+ 2456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:31.895932 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:31.895945 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:31.896053 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:31.896977 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:31.897014 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.897987 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.897995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:31.900056 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:31.900073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:31.900845 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:31.900854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:31.901177 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:31.901923 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:31.901926 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:31.901963 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:31.902040 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:31.902412 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:31.902423 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:31.902432 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 11:45:31.902613 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 11:45:31.903004 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":299,"threads_started":1}
I20260924 11:45:31.905491 20533 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37253
I20260924 11:45:31.905795 20533 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:31.906471 20533 heartbeater.cc:511] Master 127.0.97.62:37253 requested a full tablet report, sending...
I20260924 11:45:31.907825 20216 ts_manager.cc:194] Registered new tserver with Master: f604a589343b4d3b97f2a5f574ed5b46 (127.0.97.2:46151)
I20260924 11:45:31.908629 20216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:46905
W20260924 11:45:32.216060 20536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:32.216403 20536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:32.216573 20536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:32.223634 20536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:32.223755 20536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:32.223922 20536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 11:45:32.232462 20536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:32.233709 20536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:32.235361 20536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:32.247290 20544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:32.247336 20542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:32.247363 20541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:32.248004 20536 server_base.cc:1034] running on GCE node
I20260924 11:45:32.248615 20536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:32.249418 20536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:32.250645 20536 hybrid_clock.cc:648] HybridClock initialized: now 1790250332250605 us; error 53 us; skew 500 ppm
I20260924 11:45:32.250913 20536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:45:32.253933 20536 webserver.cc:460] Webserver started at http://127.0.97.3:42275/ using document root <none> and password file <none>
I20260924 11:45:32.254575 20536 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:32.254693 20536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:32.254987 20536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:32.257627 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data/instance:
uuid: "4dce3e29f7e544978e41fecb150b751d"
format_stamp: "Formatted at 2026-09-24 11:45:32 on dist-test-slave-qhrh"
server_key: "ac5d372d6eb95e2535f933a9f28aee8a"
server_key_iv: "6b78909dd3531b117c6929f81ee4d7b3"
server_key_version: "encryptionkey@0"
I20260924 11:45:32.258361 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4dce3e29f7e544978e41fecb150b751d"
format_stamp: "Formatted at 2026-09-24 11:45:32 on dist-test-slave-qhrh"
server_key: "ac5d372d6eb95e2535f933a9f28aee8a"
server_key_iv: "6b78909dd3531b117c6929f81ee4d7b3"
server_key_version: "encryptionkey@0"
I20260924 11:45:32.262791 20536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:45:32.265955 20550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:32.267077 20536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:32.267251 20536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal
uuid: "4dce3e29f7e544978e41fecb150b751d"
format_stamp: "Formatted at 2026-09-24 11:45:32 on dist-test-slave-qhrh"
server_key: "ac5d372d6eb95e2535f933a9f28aee8a"
server_key_iv: "6b78909dd3531b117c6929f81ee4d7b3"
server_key_version: "encryptionkey@0"
I20260924 11:45:32.267432 20536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:32.291627 20536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:32.292421 20536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:32.292698 20536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:32.293648 20536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:45:32.295024 20536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:45:32.295137 20536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:32.295269 20536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:45:32.295352 20536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:32.325923 20536 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34627
I20260924 11:45:32.325973 20662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34627 every 8 connection(s)
I20260924 11:45:32.327126 20536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/data/info.pb
I20260924 11:45:32.329301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20536
I20260924 11:45:32.329483 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal/instance
I20260924 11:45:32.329995 388 external_mini_cluster.cc:1428] Setting key 86771d074493740f1fd31983d8a0c4a0
I20260924 11:45:32.339329 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.329500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54965 (local address 127.0.97.62:37253)
0924 11:45:32.329720 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.329727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.331239 (+ 1512us) server_negotiation.cc:407] Connection header received
0924 11:45:32.333268 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.333272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.333335 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.333411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:32.334340 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.335001 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.336492 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.336848 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.336934 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.338757 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:32.338760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:32.338764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:32.338778 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:32.338803 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:32.338812 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:32.338894 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.339042 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.339109 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 11:45:32.340229 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.330008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37253 (local address 127.0.97.3:54965)
0924 11:45:32.331065 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.331114 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:32.333037 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:32.333580 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:32.333587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:32.333667 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:45:32.334199 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.334214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.335126 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.335129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.336382 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.336393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.337007 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.337011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.337176 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.338510 (+ 1334us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:45:32.338512 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:45:32.338553 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:45:32.338616 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:45:32.338893 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:45:32.338897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:45:32.338901 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 11:45:32.339020 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 11:45:32.339248 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":341,"threads_started":1}
I20260924 11:45:32.341408 20663 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37253
I20260924 11:45:32.341701 20663 heartbeater.cc:464] Registering TS with master...
I20260924 11:45:32.342332 20663 heartbeater.cc:511] Master 127.0.97.62:37253 requested a full tablet report, sending...
I20260924 11:45:32.343740 20216 ts_manager.cc:194] Registered new tserver with Master: 4dce3e29f7e544978e41fecb150b751d (127.0.97.3:34627)
I20260924 11:45:32.344614 20216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54965
I20260924 11:45:32.344875 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:45:32.353761 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.348406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.97.62:37253)
0924 11:45:32.348575 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.348579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.348914 (+ 335us) server_negotiation.cc:407] Connection header received
0924 11:45:32.349031 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.349034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.349090 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.349159 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:32.349587 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.350191 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.351356 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.351679 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.351758 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.353250 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:32.353254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:32.353257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:32.353271 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:32.353298 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:32.353308 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:32.353375 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.353485 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.353555 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:45:32.357925 20216 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:45:32.364287 20216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:45:32.390681 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.375222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.0.97.3:34627)
0924 11:45:32.375965 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.375975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.376699 (+ 724us) server_negotiation.cc:407] Connection header received
0924 11:45:32.376796 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.376803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.377027 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.377187 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:32.379247 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.380995 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.385285 (+ 4290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.388041 (+ 2756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.388814 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.389149 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.389903 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.390125 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":398,"threads_started":1}
I20260924 11:45:32.391031 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.374982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34627 (local address 127.0.0.1:42244)
0924 11:45:32.376245 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.376328 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:32.376501 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:32.377507 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:32.377517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:32.377552 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:32.379046 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.379059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.381172 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.381180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.385071 (+ 3891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.385089 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.388313 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.388325 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.388614 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.388663 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 11:45:32.389288 (+ 625us) client_negotiation.cc:240] Negotiation successful
0924 11:45:32.389523 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":888,"threads_started":1}
I20260924 11:45:32.395077 20598 tablet_service.cc:1467] Processing CreateTablet for tablet e01a8fe1a9734d6bbafbc84ee2a33196 (DEFAULT_TABLE table=test-table [id=93cd259cb6534d478ab2fa5e5c178a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:32.396301 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.376381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46151 (local address 127.0.0.1:59244)
0924 11:45:32.388584 (+ 12203us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.388603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:32.388713 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:32.390103 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:32.390106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:32.390124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:32.390353 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.390359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.391929 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.391933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.393947 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.393961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.395544 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.395551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.395742 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.395777 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:45:32.395874 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:45:32.395996 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"thread_start_us":12682,"threads_started":1}
I20260924 11:45:32.396448 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.376749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.0.97.2:46151)
0924 11:45:32.388845 (+ 12096us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.388856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.388892 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:45:32.389396 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.389404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.389628 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.389789 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:32.390564 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.391787 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.394234 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.395332 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.395478 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.395756 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.395928 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.396091 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11910,"thread_start_us":11790,"threads_started":1}
I20260924 11:45:32.397269 20598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01a8fe1a9734d6bbafbc84ee2a33196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:32.399566 20468 tablet_service.cc:1467] Processing CreateTablet for tablet e01a8fe1a9734d6bbafbc84ee2a33196 (DEFAULT_TABLE table=test-table [id=93cd259cb6534d478ab2fa5e5c178a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:32.400811 20468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01a8fe1a9734d6bbafbc84ee2a33196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:32.402738 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.376666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42888)
0924 11:45:32.389290 (+ 12624us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.389307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:32.389393 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:32.395103 (+ 5710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:32.395108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:32.395133 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:32.395485 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.395498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.398365 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.398369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.400204 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.400222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.402012 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.402018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.402220 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.402251 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:45:32.402343 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 11:45:32.402448 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12502,"thread_start_us":11617,"threads_started":1}
I20260924 11:45:32.402889 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.377119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.0.97.1:32975)
0924 11:45:32.394551 (+ 17432us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.394559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.394587 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:45:32.394683 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.394689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.394899 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.395103 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:32.396004 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.398230 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.400429 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.401821 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.401951 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.402162 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.402400 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.402567 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17282,"thread_start_us":17170,"threads_started":1}
I20260924 11:45:32.406208 20337 tablet_service.cc:1467] Processing CreateTablet for tablet e01a8fe1a9734d6bbafbc84ee2a33196 (DEFAULT_TABLE table=test-table [id=93cd259cb6534d478ab2fa5e5c178a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:45:32.407418 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01a8fe1a9734d6bbafbc84ee2a33196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:32.410506 20681 tablet_bootstrap.cc:492] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Bootstrap starting.
I20260924 11:45:32.411882 20682 tablet_bootstrap.cc:492] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Bootstrap starting.
I20260924 11:45:32.414392 20681 tablet_bootstrap.cc:654] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:32.415084 20682 tablet_bootstrap.cc:654] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:32.415436 20681 log.cc:824] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:32.416198 20682 log.cc:824] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:32.417994 20681 tablet_bootstrap.cc:492] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: No bootstrap required, opened a new log
I20260924 11:45:32.418259 20681 ts_tablet_manager.cc:1397] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260924 11:45:32.422708 20682 tablet_bootstrap.cc:492] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: No bootstrap required, opened a new log
I20260924 11:45:32.423048 20682 ts_tablet_manager.cc:1397] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.011s
I20260924 11:45:32.423614 20685 tablet_bootstrap.cc:492] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Bootstrap starting.
I20260924 11:45:32.426841 20685 tablet_bootstrap.cc:654] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:32.427269 20681 raft_consensus.cc:348] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.427671 20681 raft_consensus.cc:374] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:32.427778 20681 raft_consensus.cc:729] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce3e29f7e544978e41fecb150b751d, State: Initialized, Role: FOLLOWER
I20260924 11:45:32.427947 20685 log.cc:824] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:32.428509 20681 consensus_queue.cc:260] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.429664 20663 heartbeater.cc:503] Master 127.0.97.62:37253 was elected leader, sending a full tablet report...
I20260924 11:45:32.430522 20681 ts_tablet_manager.cc:1428] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260924 11:45:32.431264 20682 raft_consensus.cc:348] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.431602 20682 raft_consensus.cc:374] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:32.431716 20682 raft_consensus.cc:729] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f604a589343b4d3b97f2a5f574ed5b46, State: Initialized, Role: FOLLOWER
I20260924 11:45:32.432294 20682 consensus_queue.cc:260] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.433213 20685 tablet_bootstrap.cc:492] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: No bootstrap required, opened a new log
I20260924 11:45:32.433504 20685 ts_tablet_manager.cc:1397] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 11:45:32.433620 20533 heartbeater.cc:503] Master 127.0.97.62:37253 was elected leader, sending a full tablet report...
I20260924 11:45:32.434057 20682 ts_tablet_manager.cc:1428] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260924 11:45:32.440793 20685 raft_consensus.cc:348] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.441130 20685 raft_consensus.cc:374] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:32.441201 20685 raft_consensus.cc:729] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68a0674765b4318be98a6728543c346, State: Initialized, Role: FOLLOWER
I20260924 11:45:32.441792 20685 consensus_queue.cc:260] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.442783 20402 heartbeater.cc:503] Master 127.0.97.62:37253 was elected leader, sending a full tablet report...
I20260924 11:45:32.443228 20685 ts_tablet_manager.cc:1428] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260924 11:45:32.473531 20403 tablet.cc:2369] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:45:32.578859 20664 tablet.cc:2369] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:45:32.640331 20534 tablet.cc:2369] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:45:32.651364 20688 raft_consensus.cc:482] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:45:32.651669 20688 raft_consensus.cc:504] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.653249 20688 leader_election.cc:288] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dce3e29f7e544978e41fecb150b751d (127.0.97.3:34627), f68a0674765b4318be98a6728543c346 (127.0.97.1:32975)
I20260924 11:45:32.661154 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.654460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.97.2:41761)
0924 11:45:32.655257 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.655300 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:32.655436 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:32.655818 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:32.655822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:32.655845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:32.656233 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.656245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.657755 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.657761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.659236 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.659246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.660300 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.660306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.660514 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.660555 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:45:32.660679 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 11:45:32.660815 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":2,"thread_start_us":511,"threads_started":1}
I20260924 11:45:32.661257 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.653638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57635 (local address 127.0.97.3:34627)
0924 11:45:32.653809 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.653832 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.654378 (+ 546us) server_negotiation.cc:407] Connection header received
0924 11:45:32.654608 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.654616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.654693 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.654751 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:32.655638 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.656672 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.658591 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.660084 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.660275 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.660418 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.660844 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.660996 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:45:32.661312 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.654759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41761 (local address 127.0.97.1:32975)
0924 11:45:32.654952 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.654960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.655400 (+ 440us) server_negotiation.cc:407] Connection header received
0924 11:45:32.655596 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.655603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.655692 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.655797 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:45:32.656396 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.657600 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.659406 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.660107 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.660257 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.660346 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.660732 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.660919 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 11:45:32.662051 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.653623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34627 (local address 127.0.97.2:57635)
0924 11:45:32.654237 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.654251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:45:32.654417 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:45:32.654945 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:45:32.654950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:45:32.655001 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:45:32.655448 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.655458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.656787 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.656794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.658372 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:45:32.658385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.660470 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.660476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:45:32.660687 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.660713 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:45:32.661586 (+ 873us) client_negotiation.cc:240] Negotiation successful
0924 11:45:32.661730 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":24,"thread_start_us":310,"threads_started":1}
I20260924 11:45:32.662606 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01a8fe1a9734d6bbafbc84ee2a33196" candidate_uuid: "f604a589343b4d3b97f2a5f574ed5b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68a0674765b4318be98a6728543c346" is_pre_election: true
I20260924 11:45:32.663106 20618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01a8fe1a9734d6bbafbc84ee2a33196" candidate_uuid: "f604a589343b4d3b97f2a5f574ed5b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce3e29f7e544978e41fecb150b751d" is_pre_election: true
I20260924 11:45:32.663234 20357 raft_consensus.cc:2454] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f604a589343b4d3b97f2a5f574ed5b46 in term 0.
I20260924 11:45:32.663558 20618 raft_consensus.cc:2454] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f604a589343b4d3b97f2a5f574ed5b46 in term 0.
I20260924 11:45:32.663990 20421 leader_election.cc:302] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f604a589343b4d3b97f2a5f574ed5b46, f68a0674765b4318be98a6728543c346; no voters:
I20260924 11:45:32.664394 20688 raft_consensus.cc:2781] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:45:32.664528 20688 raft_consensus.cc:482] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:45:32.664619 20688 raft_consensus.cc:3037] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:32.666738 20688 raft_consensus.cc:504] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.667227 20688 leader_election.cc:288] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [CANDIDATE]: Term 1 election: Requested vote from peers 4dce3e29f7e544978e41fecb150b751d (127.0.97.3:34627), f68a0674765b4318be98a6728543c346 (127.0.97.1:32975)
I20260924 11:45:32.667580 20618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01a8fe1a9734d6bbafbc84ee2a33196" candidate_uuid: "f604a589343b4d3b97f2a5f574ed5b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce3e29f7e544978e41fecb150b751d"
I20260924 11:45:32.667776 20618 raft_consensus.cc:3037] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:32.668329 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01a8fe1a9734d6bbafbc84ee2a33196" candidate_uuid: "f604a589343b4d3b97f2a5f574ed5b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68a0674765b4318be98a6728543c346"
I20260924 11:45:32.668529 20357 raft_consensus.cc:3037] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:32.670797 20357 raft_consensus.cc:2454] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f604a589343b4d3b97f2a5f574ed5b46 in term 1.
I20260924 11:45:32.671205 20421 leader_election.cc:302] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f604a589343b4d3b97f2a5f574ed5b46, f68a0674765b4318be98a6728543c346; no voters:
I20260924 11:45:32.671473 20688 raft_consensus.cc:2781] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:32.671542 20618 raft_consensus.cc:2454] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f604a589343b4d3b97f2a5f574ed5b46 in term 1.
I20260924 11:45:32.672073 20688 raft_consensus.cc:686] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 1 LEADER]: Becoming Leader. State: Replica: f604a589343b4d3b97f2a5f574ed5b46, State: Running, Role: LEADER
I20260924 11:45:32.672612 20688 consensus_queue.cc:237] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } }
I20260924 11:45:32.676928 20215 catalog_manager.cc:5495] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 reported cstate change: term changed from 0 to 1, leader changed from <none> to f604a589343b4d3b97f2a5f574ed5b46 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f604a589343b4d3b97f2a5f574ed5b46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f604a589343b4d3b97f2a5f574ed5b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260924 11:45:32.746644 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.742054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.0.97.2:46151)
0924 11:45:32.742221 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.742225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.742240 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:45:32.742322 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.742325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.742404 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.742546 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:45:32.742879 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.743573 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.744828 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.745226 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.745362 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.745760 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:45:32.746207 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:45:32.746317 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.746385 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.746436 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:32.754475 20357 raft_consensus.cc:1261] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 1 FOLLOWER]: Refusing update from remote peer f604a589343b4d3b97f2a5f574ed5b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:32.754477 20618 raft_consensus.cc:1261] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 1 FOLLOWER]: Refusing update from remote peer f604a589343b4d3b97f2a5f574ed5b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:45:32.755439 20692 consensus_queue.cc:1035] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dce3e29f7e544978e41fecb150b751d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:32.755754 20688 consensus_queue.cc:1035] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68a0674765b4318be98a6728543c346" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:45:32.767093 20695 mvcc.cc:205] Tried to move back new op lower bound from 7332865362952257536 to 7332865362631106560. Current Snapshot: MvccSnapshot[applied={T|T < 7332865362952257536}]
I20260924 11:45:32.778329 20214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:32.778607 20214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:45:32.782146 20214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f6bf48d023f452488df1fd2a6d9d1d0: Sending DeleteTablet for 3 replicas of tablet e01a8fe1a9734d6bbafbc84ee2a33196
I20260924 11:45:32.783365 20337 tablet_service.cc:1514] Processing DeleteTablet for tablet e01a8fe1a9734d6bbafbc84ee2a33196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:32 UTC) from {username='slave'} at 127.0.0.1:42888
I20260924 11:45:32.783622 20468 tablet_service.cc:1514] Processing DeleteTablet for tablet e01a8fe1a9734d6bbafbc84ee2a33196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:32 UTC) from {username='slave'} at 127.0.0.1:59244
I20260924 11:45:32.784370 20598 tablet_service.cc:1514] Processing DeleteTablet for tablet e01a8fe1a9734d6bbafbc84ee2a33196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:45:32 UTC) from {username='slave'} at 127.0.0.1:42244
I20260924 11:45:32.785706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20275
I20260924 11:45:32.785822 20706 tablet_replica.cc:330] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: stopping tablet replica
I20260924 11:45:32.786302 20706 raft_consensus.cc:2229] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:45:32.786955 20706 raft_consensus.cc:2258] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:32.787602 20707 tablet_replica.cc:330] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: stopping tablet replica
I20260924 11:45:32.787946 20707 raft_consensus.cc:2229] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:32.788384 20707 raft_consensus.cc:2258] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:32.788951 20705 tablet_replica.cc:330] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: stopping tablet replica
I20260924 11:45:32.789427 20705 raft_consensus.cc:2229] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:45:32.789923 20705 raft_consensus.cc:2258] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:45:32.790230 20706 ts_tablet_manager.cc:1905] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:32.792024 20705 ts_tablet_manager.cc:1905] T e01a8fe1a9734d6bbafbc84ee2a33196 P f68a0674765b4318be98a6728543c346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:32.796339 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.787009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.0.97.1:32975)
0924 11:45:32.787170 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.787176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.787840 (+ 664us) server_negotiation.cc:407] Connection header received
0924 11:45:32.788125 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.788130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.788247 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.788319 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:32.789099 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.790196 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.791911 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.792558 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.792662 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.794824 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:32.794836 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:32.794853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:32.795482 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:32.795531 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:32.795552 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:32.795646 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.795879 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.795984 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:45:32.797506 20397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:32.797633 20706 ts_tablet_manager.cc:1918] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:32.797782 20706 log.cc:1196] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-1/wal/wals/e01a8fe1a9734d6bbafbc84ee2a33196
I20260924 11:45:32.798084 20707 ts_tablet_manager.cc:1905] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:45:32.798542 20706 ts_tablet_manager.cc:1939] T e01a8fe1a9734d6bbafbc84ee2a33196 P f604a589343b4d3b97f2a5f574ed5b46: Deleting consensus metadata
I20260924 11:45:32.800513 20202 catalog_manager.cc:4841] TS f604a589343b4d3b97f2a5f574ed5b46 (127.0.97.2:46151): tablet e01a8fe1a9734d6bbafbc84ee2a33196 (table test-table [id=93cd259cb6534d478ab2fa5e5c178a3f]) successfully deleted
I20260924 11:45:32.805480 20707 ts_tablet_manager.cc:1918] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:45:32.805650 20707 log.cc:1196] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Deleting WAL directory at /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250027667721-388-0/minicluster-data/ts-2/wal/wals/e01a8fe1a9734d6bbafbc84ee2a33196
I20260924 11:45:32.806105 20707 ts_tablet_manager.cc:1939] T e01a8fe1a9734d6bbafbc84ee2a33196 P 4dce3e29f7e544978e41fecb150b751d: Deleting consensus metadata
I20260924 11:45:32.807678 20200 catalog_manager.cc:4841] TS 4dce3e29f7e544978e41fecb150b751d (127.0.97.3:34627): tablet e01a8fe1a9734d6bbafbc84ee2a33196 (table test-table [id=93cd259cb6534d478ab2fa5e5c178a3f]) successfully deleted
I20260924 11:45:32.886240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20275
I20260924 11:45:32.910247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20406
W20260924 11:45:32.910166 20200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32975 (error 108)
I20260924 11:45:32.920535 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.910923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.0.97.2:46151)
0924 11:45:32.911112 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:32.911122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:32.911329 (+ 207us) server_negotiation.cc:407] Connection header received
0924 11:45:32.911393 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:32.911397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:32.911478 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:32.911586 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:32.912250 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.913476 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.915340 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:32.915988 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:32.916099 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:32.918611 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:32.918622 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:32.918640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:32.919365 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:32.919419 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:32.919442 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:32.919536 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:32.920048 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 11:45:32.920178 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:32.921451 20528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:32.925645 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.923998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42904)
0924 11:45:32.925221 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.925395 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":919,"threads_started":1}
W20260924 11:45:32.925949 20200 catalog_manager.cc:4584] TS f68a0674765b4318be98a6728543c346 (127.0.97.1:32975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e01a8fe1a9734d6bbafbc84ee2a33196: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
I20260924 11:45:32.979501 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.979035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42914)
0924 11:45:32.979298 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.979379 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260924 11:45:32.980602 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:32.980274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42922)
0924 11:45:32.980420 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:32.980486 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 11:45:33.033110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20406
I20260924 11:45:33.041152 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:33.040731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42934)
0924 11:45:33.040927 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:33.041009 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 11:45:33.042119 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:33.041799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42946)
0924 11:45:33.041937 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:33.042003 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 11:45:33.052003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20536
I20260924 11:45:33.067814 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:33.052736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.97.3:34627)
0924 11:45:33.052910 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:33.052915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:33.052948 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:45:33.053121 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:33.053125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:33.053211 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:33.053291 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:33.053619 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:33.054306 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:33.055515 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:33.055866 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:33.055947 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:33.066560 (+ 10613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:33.066568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:33.066591 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:33.067147 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:33.067191 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:33.067209 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:33.067300 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:33.067391 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 11:45:33.067471 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:45:33.068507 20658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:33.122649 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:33.122199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42958)
0924 11:45:33.122434 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:33.122514 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 11:45:33.123736 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:33.123364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32975 (local address 127.0.0.1:42960)
0924 11:45:33.123559 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 11:45:33.123620 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 11:45:33.151732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20536
I20260924 11:45:33.167585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20185
I20260924 11:45:33.168643 20246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:33.271823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20185
2026-09-24T11:45:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2796 ms)
[----------] 6 tests from EncryptionPolicyTest (20347 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:33 dist-test-slave-qhrh krb5kdc[20717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:33 dist-test-slave-qhrh krb5kdc[20717](info): set up 2 sockets
Sep 24 11:45:33 dist-test-slave-qhrh krb5kdc[20717](info): commencing operation
krb5kdc: starting...
W20260924 11:45:34.312587 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:34 dist-test-slave-qhrh krb5kdc[20717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:34.535669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:34.835870 20730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:34.836122 20730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:34.836318 20730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:34.843686 20730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:34.843814 20730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:34.843895 20730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:34.843998 20730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:34.844077 20730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:34.853375 20730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:34.854565 20730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:34.855816 20730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:34.867048 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:34.867105 20735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:34.867128 20736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:34.868346 20730 server_base.cc:1034] running on GCE node
I20260924 11:45:34.869086 20730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:34.870404 20730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:34.871595 20730 hybrid_clock.cc:648] HybridClock initialized: now 1790250334871556 us; error 67 us; skew 500 ppm
I20260924 11:45:34.871932 20730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:34 dist-test-slave-qhrh krb5kdc[20717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:34.875097 20730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:34.876597 20730 webserver.cc:460] Webserver started at http://127.0.0.1:37333/ using document root <none> and password file <none>
I20260924 11:45:34.877242 20730 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:34.877364 20730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:34.877669 20730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:34.880375 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "33fd689e7b3941c5b2bcdba2e73d9dde"
format_stamp: "Formatted at 2026-09-24 11:45:34 on dist-test-slave-qhrh"
server_key: "640f3ff1846615ad6265a1acd7894866"
server_key_iv: "5271f50e187b98791fd2466861dc857f"
server_key_version: "encryptionkey@0"
I20260924 11:45:34.881088 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "33fd689e7b3941c5b2bcdba2e73d9dde"
format_stamp: "Formatted at 2026-09-24 11:45:34 on dist-test-slave-qhrh"
server_key: "640f3ff1846615ad6265a1acd7894866"
server_key_iv: "5271f50e187b98791fd2466861dc857f"
server_key_version: "encryptionkey@0"
I20260924 11:45:34.885882 20730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:34.889107 20745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:34.890275 20730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:34.890440 20730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "33fd689e7b3941c5b2bcdba2e73d9dde"
format_stamp: "Formatted at 2026-09-24 11:45:34 on dist-test-slave-qhrh"
server_key: "640f3ff1846615ad6265a1acd7894866"
server_key_iv: "5271f50e187b98791fd2466861dc857f"
server_key_version: "encryptionkey@0"
I20260924 11:45:34.890622 20730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:34.941892 20730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:34.946123 20730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:34.946417 20730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:34.964531 20730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43605
I20260924 11:45:34.964588 20796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43605 every 8 connection(s)
I20260924 11:45:34.965850 20730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:34.967136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20730
I20260924 11:45:34.967357 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:34.968024 388 external_mini_cluster.cc:1428] Setting key 4e2515dbae4c3f87484f8b86fda3624c
I20260924 11:45:34.970280 20797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:45:34 dist-test-slave-qhrh krb5kdc[20717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:45:34.982810 20797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde: Bootstrap starting.
I20260924 11:45:34.986693 20797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:34.987805 20797 log.cc:824] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:34.990617 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:34.969915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57707 (local address 127.0.0.1:43605)
0924 11:45:34.970708 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:34.970719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:34.970758 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:45:34.972364 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:34.972383 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:34.972723 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:34.973031 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:34.974831 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:34.976130 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:34.977832 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:34.978403 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:34.978528 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:34.982843 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:34.982853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:34.982870 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:34.982916 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:34.986499 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:34.987203 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:34.987210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:34.987216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:34.987391 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:34.987740 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:34.987746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:34.987749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:34.988420 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:34.988683 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:34.989187 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 11:45:34.989461 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":410,"threads_started":1}
I20260924 11:45:34.990799 20797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde: No bootstrap required, opened a new log
I20260924 11:45:34.994446 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:34.998296 20797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260924 11:45:34.998582 20797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:34.998732 20797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fd689e7b3941c5b2bcdba2e73d9dde, State: Initialized, Role: FOLLOWER
I20260924 11:45:34.999251 20797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260924 11:45:34.999449 20797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:34.999560 20797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:34.999703 20797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:35.001993 20797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260924 11:45:35.002435 20797 leader_election.cc:302] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33fd689e7b3941c5b2bcdba2e73d9dde; no voters:
I20260924 11:45:35.003093 20797 leader_election.cc:288] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:35.003304 20806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:35.006300 20806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [term 1 LEADER]: Becoming Leader. State: Replica: 33fd689e7b3941c5b2bcdba2e73d9dde, State: Running, Role: LEADER
I20260924 11:45:35.006872 20806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260924 11:45:35.007030 20797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:35.010072 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } } }
I20260924 11:45:35.010447 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:35.014359 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33fd689e7b3941c5b2bcdba2e73d9dde. Latest consensus state: current_term: 1 leader_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fd689e7b3941c5b2bcdba2e73d9dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } } }
I20260924 11:45:35.014772 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:35.016095 20813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:35.025364 20813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:35.027618 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:35.002908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50623 (local address 127.0.0.1:43605)
0924 11:45:35.003085 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:35.003091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:35.004349 (+ 1258us) server_negotiation.cc:407] Connection header received
0924 11:45:35.004541 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:35.004548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:35.004630 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:35.004725 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:35.006146 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:35.007177 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:35.008730 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:35.009395 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:35.014541 (+ 5146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:35.018154 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:35.018174 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:35.018179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:35.018225 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:35.023489 (+ 5264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:35.024484 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:35.024490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:35.024493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:35.024676 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:35.025253 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:35.025258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:35.025261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:35.025632 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:35.025923 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:35.027111 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0924 11:45:35.027300 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:45:35.035792 20813 catalog_manager.cc:1347] Generated new cluster ID: e4ba3fe281d741b999977541f27291a8
I20260924 11:45:35.035892 20813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:35.042866 20813 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:35.044554 20813 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:35.057611 20813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33fd689e7b3941c5b2bcdba2e73d9dde: Generated new TSK 0
I20260924 11:45:35.058480 20813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:35.067616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20730
I20260924 11:45:35.068780 20792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:35.153424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20730
W20260924 11:45:35.170097 20715 connection.cc:535] client connection to 127.0.0.1:43605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43605): Connection reset by peer (error 104)
2026-09-24T11:45:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:35 dist-test-slave-qhrh krb5kdc[20830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:35 dist-test-slave-qhrh krb5kdc[20830](info): set up 2 sockets
Sep 24 11:45:35 dist-test-slave-qhrh krb5kdc[20830](info): commencing operation
krb5kdc: starting...
W20260924 11:45:36.188278 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:36 dist-test-slave-qhrh krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:36.410466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:36.718169 20843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:36.718474 20843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:36.718631 20843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:36.725730 20843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:36.725859 20843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:36.725991 20843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:36.726104 20843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:36.726179 20843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:36.735517 20843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:36.736707 20843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:36.737890 20843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:36.749258 20849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:36.749297 20851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:36.749382 20848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:36.749985 20843 server_base.cc:1034] running on GCE node
I20260924 11:45:36.750669 20843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:36.751963 20843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:36.753180 20843 hybrid_clock.cc:648] HybridClock initialized: now 1790250336753127 us; error 47 us; skew 500 ppm
I20260924 11:45:36.753526 20843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:36 dist-test-slave-qhrh krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:36.756484 20843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:36.757833 20843 webserver.cc:460] Webserver started at http://127.0.0.1:35911/ using document root <none> and password file <none>
I20260924 11:45:36.758425 20843 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:36.758559 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:36.758843 20843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:36.761669 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "e53177973f8f42228941af8d65dab904"
format_stamp: "Formatted at 2026-09-24 11:45:36 on dist-test-slave-qhrh"
server_key: "45a142038afb48b42ad924ddab55521b"
server_key_iv: "d7a4bd38d409324fd0a7eef3d9bad298"
server_key_version: "encryptionkey@0"
I20260924 11:45:36.762406 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "e53177973f8f42228941af8d65dab904"
format_stamp: "Formatted at 2026-09-24 11:45:36 on dist-test-slave-qhrh"
server_key: "45a142038afb48b42ad924ddab55521b"
server_key_iv: "d7a4bd38d409324fd0a7eef3d9bad298"
server_key_version: "encryptionkey@0"
I20260924 11:45:36.767195 20843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:45:36.770282 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:36.771364 20843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:36.771579 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "e53177973f8f42228941af8d65dab904"
format_stamp: "Formatted at 2026-09-24 11:45:36 on dist-test-slave-qhrh"
server_key: "45a142038afb48b42ad924ddab55521b"
server_key_iv: "d7a4bd38d409324fd0a7eef3d9bad298"
server_key_version: "encryptionkey@0"
I20260924 11:45:36.771762 20843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:36.789127 20843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:36.792685 20843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:36.792963 20843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:36.810580 20843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39265
I20260924 11:45:36.810616 20909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39265 every 8 connection(s)
I20260924 11:45:36.811856 20843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:36.815150 20910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:36.821038 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20843
I20260924 11:45:36.821221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:36.821683 388 external_mini_cluster.cc:1428] Setting key 6f8b6829a0d1629e00f30ef7817f7831
I20260924 11:45:36.825145 20910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904: Bootstrap starting.
I20260924 11:45:36.828027 20910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:36.829017 20910 log.cc:824] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:36.831475 20910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904: No bootstrap required, opened a new log
I20260924 11:45:36.834002 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:36.823324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55947 (local address 127.0.0.1:39265)
0924 11:45:36.824051 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:36.824061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:36.824099 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:45:36.825816 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:36.825837 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:36.826171 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:36.826536 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:36.827073 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:36.828400 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:36.829803 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:36.830438 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:36.830630 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:36.831821 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:36.831828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:36.831840 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:36.832417 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:36.832488 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:36.832514 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:36.832665 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:36.832957 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 11:45:36.833104 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":323,"threads_started":1}
I20260924 11:45:36.837570 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:36.839046 20910 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53177973f8f42228941af8d65dab904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260924 11:45:36.839370 20910 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:36.839466 20910 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53177973f8f42228941af8d65dab904, State: Initialized, Role: FOLLOWER
I20260924 11:45:36.840021 20910 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53177973f8f42228941af8d65dab904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260924 11:45:36.840283 20910 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:36.840413 20910 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:36.840581 20910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:36.843328 20910 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53177973f8f42228941af8d65dab904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260924 11:45:36.843832 20910 leader_election.cc:302] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53177973f8f42228941af8d65dab904; no voters:
I20260924 11:45:36.844513 20910 leader_election.cc:288] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:36.844625 20920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:36.846239 20920 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [term 1 LEADER]: Becoming Leader. State: Replica: e53177973f8f42228941af8d65dab904, State: Running, Role: LEADER
I20260924 11:45:36.846751 20910 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:36.846815 20920 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53177973f8f42228941af8d65dab904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260924 11:45:36.848604 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:36.841605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60885 (local address 127.0.0.1:39265)
0924 11:45:36.841859 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:36.841867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:36.842393 (+ 526us) server_negotiation.cc:407] Connection header received
0924 11:45:36.842644 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:36.842650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:36.842749 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:36.842845 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:36.843370 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:36.844364 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:36.846039 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:36.846630 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:36.846771 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:36.847834 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:36.847840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:36.847845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:36.847870 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:36.847925 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:36.847940 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:36.848085 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:36.848296 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 11:45:36.848345 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:45:36.851011 20921 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53177973f8f42228941af8d65dab904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53177973f8f42228941af8d65dab904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } } }
I20260924 11:45:36.851424 20921 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:36.856346 20922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53177973f8f42228941af8d65dab904. Latest consensus state: current_term: 1 leader_uuid: "e53177973f8f42228941af8d65dab904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53177973f8f42228941af8d65dab904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } } }
I20260924 11:45:36.856591 20922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:36.857131 20928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:36.861543 20928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:36.869282 20928 catalog_manager.cc:1347] Generated new cluster ID: 8fc45bd010b9445aa75cd243fb7e791e
I20260924 11:45:36.869377 20928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:36.906105 20928 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:36.907352 20928 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:36.917194 20928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53177973f8f42228941af8d65dab904: Generated new TSK 0
I20260924 11:45:36.918186 20928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:36.934877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20843
W20260924 11:45:36.935849 20905 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55947
W20260924 11:45:36.936609 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20843: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:36.936801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20843
W20260924 11:45:36.952988 20828 connection.cc:535] client connection to 127.0.0.1:39265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39265): Connection reset by peer (error 104)
2026-09-24T11:45:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1784 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:37 dist-test-slave-qhrh krb5kdc[20941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:37 dist-test-slave-qhrh krb5kdc[20941](info): set up 2 sockets
Sep 24 11:45:37 dist-test-slave-qhrh krb5kdc[20941](info): commencing operation
krb5kdc: starting...
W20260924 11:45:37.972704 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:38 dist-test-slave-qhrh krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:38.194763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:38.496157 20954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:38.496449 20954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:38.496603 20954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:38.503506 20954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:38.503654 20954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:38.503765 20954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:38.503882 20954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:38.503968 20954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:38.512876 20954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:38.514065 20954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:38.515301 20954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:38.526435 20960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:38.526515 20962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:38.526464 20959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:38.527302 20954 server_base.cc:1034] running on GCE node
I20260924 11:45:38.528050 20954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:38.529379 20954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:38.530597 20954 hybrid_clock.cc:648] HybridClock initialized: now 1790250338530552 us; error 40 us; skew 500 ppm
I20260924 11:45:38.530926 20954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:38 dist-test-slave-qhrh krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:38.542189 20954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:38.543633 20954 webserver.cc:460] Webserver started at http://127.0.0.1:39735/ using document root <none> and password file <none>
I20260924 11:45:38.544418 20954 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:38.544562 20954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:38.544888 20954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:38.547689 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "5a631a90b828447bbea32c77a9341e68"
format_stamp: "Formatted at 2026-09-24 11:45:38 on dist-test-slave-qhrh"
server_key: "c83138fcea7346febcea809f8be168e8"
server_key_iv: "ece8da0f5ef060dfcd8b588c0da72a9e"
server_key_version: "encryptionkey@0"
I20260924 11:45:38.548405 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a631a90b828447bbea32c77a9341e68"
format_stamp: "Formatted at 2026-09-24 11:45:38 on dist-test-slave-qhrh"
server_key: "c83138fcea7346febcea809f8be168e8"
server_key_iv: "ece8da0f5ef060dfcd8b588c0da72a9e"
server_key_version: "encryptionkey@0"
I20260924 11:45:38.553150 20954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:45:38.556452 20969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:38.557729 20954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:45:38.557914 20954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "5a631a90b828447bbea32c77a9341e68"
format_stamp: "Formatted at 2026-09-24 11:45:38 on dist-test-slave-qhrh"
server_key: "c83138fcea7346febcea809f8be168e8"
server_key_iv: "ece8da0f5ef060dfcd8b588c0da72a9e"
server_key_version: "encryptionkey@0"
I20260924 11:45:38.558105 20954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:38.563683 20954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:38.568238 20954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:38.568532 20954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:38.586563 20954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43087
I20260924 11:45:38.586601 21020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43087 every 8 connection(s)
I20260924 11:45:38.587939 20954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:38.591405 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:38.596480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 20954
I20260924 11:45:38.596684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:38.597139 388 external_mini_cluster.cc:1428] Setting key e21b12d6c0596cd496c0aab5a1cb42c2
I20260924 11:45:38.601689 21021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68: Bootstrap starting.
I20260924 11:45:38.605132 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:38.598952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45267 (local address 127.0.0.1:43087)
0924 11:45:38.599875 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:38.599886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:38.599936 (+ 50us) server_negotiation.cc:407] Connection header received
0924 11:45:38.601719 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:38.601741 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:38.602099 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:38.602421 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:38.603565 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:38.603582 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:38.603593 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:38.603608 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:38.603680 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:38.603724 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:38.603903 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:38.604017 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:45:38.604241 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":481,"threads_started":1}
I20260924 11:45:38.605161 21021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:38.606389 21021 log.cc:824] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:38.608987 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:38.609483 21021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68: No bootstrap required, opened a new log
I20260924 11:45:38.615562 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:38.613024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49213 (local address 127.0.0.1:43087)
0924 11:45:38.613235 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:38.613243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:38.613686 (+ 443us) server_negotiation.cc:407] Connection header received
0924 11:45:38.613855 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:38.613858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:38.613899 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:38.613977 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:38.614925 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:38.614929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:38.614931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:38.614944 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:38.614981 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:38.614994 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:38.615097 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:38.615232 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 11:45:38.615272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:45:38.618122 21021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a631a90b828447bbea32c77a9341e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260924 11:45:38.618489 21021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:38.618610 21021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a631a90b828447bbea32c77a9341e68, State: Initialized, Role: FOLLOWER
I20260924 11:45:38.619208 21021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a631a90b828447bbea32c77a9341e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260924 11:45:38.619465 21021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:38.619635 21021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:38.619824 21021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:38.622385 21021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a631a90b828447bbea32c77a9341e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260924 11:45:38.622901 21021 leader_election.cc:302] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a631a90b828447bbea32c77a9341e68; no voters:
I20260924 11:45:38.623484 21021 leader_election.cc:288] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:38.623607 21031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:38.624379 21031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [term 1 LEADER]: Becoming Leader. State: Replica: 5a631a90b828447bbea32c77a9341e68, State: Running, Role: LEADER
I20260924 11:45:38.624882 21031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a631a90b828447bbea32c77a9341e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260924 11:45:38.625754 21021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:38.627744 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a631a90b828447bbea32c77a9341e68. Latest consensus state: current_term: 1 leader_uuid: "5a631a90b828447bbea32c77a9341e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a631a90b828447bbea32c77a9341e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260924 11:45:38.628129 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:38.627859 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a631a90b828447bbea32c77a9341e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a631a90b828447bbea32c77a9341e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260924 11:45:38.628319 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:38.630527 21036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:38.635670 21036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:38.643864 21036 catalog_manager.cc:1347] Generated new cluster ID: 5390a2b4c02349178f6d45d5cd64a85c
I20260924 11:45:38.643966 21036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:38.662531 21036 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:38.662946 21036 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:38.673060 21036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a631a90b828447bbea32c77a9341e68: Generated new TSK 0
I20260924 11:45:38.673946 21036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:38.682745 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20954
W20260924 11:45:38.683736 21016 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45267
W20260924 11:45:38.684428 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 20954: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:38.684577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 20954
2026-09-24T11:45:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:38 dist-test-slave-qhrh krb5kdc[21052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:38 dist-test-slave-qhrh krb5kdc[21052](info): set up 2 sockets
Sep 24 11:45:38 dist-test-slave-qhrh krb5kdc[21052](info): commencing operation
krb5kdc: starting...
W20260924 11:45:39.716390 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:39 dist-test-slave-qhrh krb5kdc[21052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:39.941154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:40.252246 21065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:40.252537 21065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:40.252709 21065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:40.259563 21065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:40.259703 21065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:40.259799 21065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:40.259891 21065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:40.259963 21065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:40.268983 21065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:40.270157 21065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:40.271451 21065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:40.282688 21071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:40.282693 21070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:40.282768 21073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:40.283378 21065 server_base.cc:1034] running on GCE node
I20260924 11:45:40.284024 21065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:40.285300 21065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:40.286543 21065 hybrid_clock.cc:648] HybridClock initialized: now 1790250340286509 us; error 46 us; skew 500 ppm
I20260924 11:45:40.286864 21065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:40 dist-test-slave-qhrh krb5kdc[21052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:40.289790 21065 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:40.291244 21065 webserver.cc:460] Webserver started at http://127.0.0.1:39715/ using document root <none> and password file <none>
I20260924 11:45:40.291846 21065 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:40.291993 21065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:40.292357 21065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:40.295104 21065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "fb74a815c51140a9be99f83b78f4f03b"
format_stamp: "Formatted at 2026-09-24 11:45:40 on dist-test-slave-qhrh"
server_key: "48f66bade6a526b2ffaabeb14c0c5367"
server_key_iv: "4726aabe9b7dd06824117933569f4c2a"
server_key_version: "encryptionkey@0"
I20260924 11:45:40.295866 21065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "fb74a815c51140a9be99f83b78f4f03b"
format_stamp: "Formatted at 2026-09-24 11:45:40 on dist-test-slave-qhrh"
server_key: "48f66bade6a526b2ffaabeb14c0c5367"
server_key_iv: "4726aabe9b7dd06824117933569f4c2a"
server_key_version: "encryptionkey@0"
I20260924 11:45:40.300554 21065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 11:45:40.303889 21080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:40.305146 21065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:45:40.305318 21065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "fb74a815c51140a9be99f83b78f4f03b"
format_stamp: "Formatted at 2026-09-24 11:45:40 on dist-test-slave-qhrh"
server_key: "48f66bade6a526b2ffaabeb14c0c5367"
server_key_iv: "4726aabe9b7dd06824117933569f4c2a"
server_key_version: "encryptionkey@0"
I20260924 11:45:40.305510 21065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:40.322384 21065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:40.325941 21065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:40.326220 21065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:40.343252 21065 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41515
I20260924 11:45:40.343287 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41515 every 8 connection(s)
I20260924 11:45:40.344591 21065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:40.347841 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:40.352054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21065
I20260924 11:45:40.352243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:40.352782 388 external_mini_cluster.cc:1428] Setting key 62dc4187cc8f0c98d580949b6626794d
I20260924 11:45:40.358207 21132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b: Bootstrap starting.
I20260924 11:45:40.361040 21132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:40.362026 21132 log.cc:824] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:40.365200 21132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b: No bootstrap required, opened a new log
Sep 24 11:45:40 dist-test-slave-qhrh krb5kdc[21052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:45:40.372417 21132 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb74a815c51140a9be99f83b78f4f03b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260924 11:45:40.372771 21132 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:40.372917 21132 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb74a815c51140a9be99f83b78f4f03b, State: Initialized, Role: FOLLOWER
I20260924 11:45:40.373524 21132 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb74a815c51140a9be99f83b78f4f03b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260924 11:45:40.373741 21132 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:40.373940 21132 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:40.374094 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:40.354475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36907 (local address 127.0.0.1:41515)
0924 11:45:40.355233 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:40.355244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:40.355278 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:45:40.357114 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:40.357132 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:40.357457 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:40.357797 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:40.359133 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:40.360400 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:40.361714 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:40.362568 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:40.362772 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:40.367059 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:40.367066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:40.367077 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:40.367123 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:40.370179 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:40.370814 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:40.370820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:40.370825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:40.370971 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:40.371334 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:40.371339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:40.371341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:40.371832 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:40.372121 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:40.372613 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 11:45:40.373081 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":359,"threads_started":1}
I20260924 11:45:40.374123 21132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:40.376904 21132 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb74a815c51140a9be99f83b78f4f03b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260924 11:45:40.377372 21132 leader_election.cc:302] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb74a815c51140a9be99f83b78f4f03b; no voters:
I20260924 11:45:40.377686 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:40.377990 21132 leader_election.cc:288] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:40.378093 21137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:40.378655 21137 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [term 1 LEADER]: Becoming Leader. State: Replica: fb74a815c51140a9be99f83b78f4f03b, State: Running, Role: LEADER
I20260924 11:45:40.379177 21137 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb74a815c51140a9be99f83b78f4f03b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260924 11:45:40.380633 21137 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb74a815c51140a9be99f83b78f4f03b. Latest consensus state: current_term: 1 leader_uuid: "fb74a815c51140a9be99f83b78f4f03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb74a815c51140a9be99f83b78f4f03b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } } }
I20260924 11:45:40.380934 21137 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:40.381359 21139 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb74a815c51140a9be99f83b78f4f03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb74a815c51140a9be99f83b78f4f03b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } } }
I20260924 11:45:40.381780 21139 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:40.389846 21132 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:40.393015 21141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:40.397280 21141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:40.415108 21141 catalog_manager.cc:1347] Generated new cluster ID: 9554d337b79a4764bf94a8c0937becaa
I20260924 11:45:40.415269 21141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:40.425879 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:40.392671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50173 (local address 127.0.0.1:41515)
0924 11:45:40.392869 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:40.392878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:40.404659 (+ 11781us) server_negotiation.cc:407] Connection header received
0924 11:45:40.405157 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:40.405164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:40.405255 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:40.405331 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:40.407650 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:40.409012 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:40.411196 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:40.411901 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:40.412014 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:40.415880 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:40.415890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:40.415897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:40.415944 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:40.420526 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:40.422846 (+ 2320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:40.422852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:40.422855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:40.423025 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:40.423523 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:40.423530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:40.423535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:40.423995 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:40.424244 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:40.425043 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 11:45:40.425336 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:45:40.433583 21141 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:40.435310 21141 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:40.456341 21141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb74a815c51140a9be99f83b78f4f03b: Generated new TSK 0
I20260924 11:45:40.457351 21141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:40.466764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21065
I20260924 11:45:40.467945 21127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:40.553108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21065
2026-09-24T11:45:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:40 dist-test-slave-qhrh krb5kdc[21164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:40 dist-test-slave-qhrh krb5kdc[21164](info): set up 2 sockets
Sep 24 11:45:40 dist-test-slave-qhrh krb5kdc[21164](info): commencing operation
krb5kdc: starting...
W20260924 11:45:41.592437 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:41 dist-test-slave-qhrh krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:41.811098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:42.120517 21177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:42.120826 21177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:42.120999 21177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:42.127899 21177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:42.128029 21177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:42.128149 21177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:42.128273 21177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:42.128353 21177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:42.137795 21177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:42.139039 21177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:42.140286 21177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:42.151315 21182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:42.151347 21185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:42.151367 21183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:42.152184 21177 server_base.cc:1034] running on GCE node
I20260924 11:45:42.152894 21177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:42.154291 21177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:42.155478 21177 hybrid_clock.cc:648] HybridClock initialized: now 1790250342155443 us; error 31 us; skew 500 ppm
I20260924 11:45:42.155793 21177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:42 dist-test-slave-qhrh krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:42.158677 21177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:42.160009 21177 webserver.cc:460] Webserver started at http://127.0.0.1:34481/ using document root <none> and password file <none>
I20260924 11:45:42.160669 21177 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:42.160805 21177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:42.161099 21177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:42.163919 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "9d35f6c50bcb4de7b83a663781da33e3"
format_stamp: "Formatted at 2026-09-24 11:45:42 on dist-test-slave-qhrh"
server_key: "8e1d126d474ae8eabdddcd09567d1738"
server_key_iv: "b9dd3b316f9238eddedb6838536fe900"
server_key_version: "encryptionkey@0"
I20260924 11:45:42.164709 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "9d35f6c50bcb4de7b83a663781da33e3"
format_stamp: "Formatted at 2026-09-24 11:45:42 on dist-test-slave-qhrh"
server_key: "8e1d126d474ae8eabdddcd09567d1738"
server_key_iv: "b9dd3b316f9238eddedb6838536fe900"
server_key_version: "encryptionkey@0"
I20260924 11:45:42.169301 21177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:45:42.172556 21192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:42.173694 21177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:42.173870 21177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "9d35f6c50bcb4de7b83a663781da33e3"
format_stamp: "Formatted at 2026-09-24 11:45:42 on dist-test-slave-qhrh"
server_key: "8e1d126d474ae8eabdddcd09567d1738"
server_key_iv: "b9dd3b316f9238eddedb6838536fe900"
server_key_version: "encryptionkey@0"
I20260924 11:45:42.174074 21177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:42.193430 21177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:42.197607 21177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:42.197865 21177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:42.220196 21177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39469
I20260924 11:45:42.220209 21243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39469 every 8 connection(s)
I20260924 11:45:42.222084 21177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:42.223086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21177
I20260924 11:45:42.223269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:42.223881 388 external_mini_cluster.cc:1428] Setting key a43738476d60c2c097f7e7237c573d12
I20260924 11:45:42.233817 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:42.250728 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3: Bootstrap starting.
I20260924 11:45:42.254006 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:42.225599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38331 (local address 127.0.0.1:39469)
0924 11:45:42.240663 (+ 15064us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:42.240679 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:42.240732 (+ 53us) server_negotiation.cc:407] Connection header received
0924 11:45:42.242566 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:42.242593 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:42.243106 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:42.243555 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:42.244629 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:42.246472 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:42.248688 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:42.249661 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:42.249823 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:42.251241 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:42.251256 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:42.251272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:42.251904 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:42.251993 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:42.252030 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:42.252165 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:42.252409 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 11:45:42.252835 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14818,"thread_start_us":14568,"threads_started":1}
I20260924 11:45:42.254630 21244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:42.255946 21244 log.cc:824] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:42.258603 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:42.258931 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3: No bootstrap required, opened a new log
I20260924 11:45:42.269289 21244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d35f6c50bcb4de7b83a663781da33e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39469 } }
I20260924 11:45:42.269758 21244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:42.269876 21244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d35f6c50bcb4de7b83a663781da33e3, State: Initialized, Role: FOLLOWER
I20260924 11:45:42.270537 21244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d35f6c50bcb4de7b83a663781da33e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39469 } }
I20260924 11:45:42.270825 21244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:42.270998 21244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:42.271458 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:42.263810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46695 (local address 127.0.0.1:39469)
0924 11:45:42.263973 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:42.263978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:42.264583 (+ 605us) server_negotiation.cc:407] Connection header received
0924 11:45:42.264656 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:42.264661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:42.264733 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:42.264824 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:42.265506 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:42.266586 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:42.268576 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:42.269179 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:42.269279 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:42.270625 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:42.270631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:42.270635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:42.270654 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:42.270696 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:42.270716 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:42.270795 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:42.271057 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 11:45:42.271139 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:45:42.271196 21244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:42.274179 21244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d35f6c50bcb4de7b83a663781da33e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39469 } }
I20260924 11:45:42.274632 21244 leader_election.cc:302] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d35f6c50bcb4de7b83a663781da33e3; no voters:
I20260924 11:45:42.275249 21244 leader_election.cc:288] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:42.275403 21254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:42.276428 21254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [term 1 LEADER]: Becoming Leader. State: Replica: 9d35f6c50bcb4de7b83a663781da33e3, State: Running, Role: LEADER
I20260924 11:45:42.277040 21254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d35f6c50bcb4de7b83a663781da33e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39469 } }
I20260924 11:45:42.277406 21244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:42.279711 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d35f6c50bcb4de7b83a663781da33e3. Latest consensus state: current_term: 1 leader_uuid: "9d35f6c50bcb4de7b83a663781da33e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d35f6c50bcb4de7b83a663781da33e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39469 } } }
I20260924 11:45:42.281143 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:42.280398 21255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d35f6c50bcb4de7b83a663781da33e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d35f6c50bcb4de7b83a663781da33e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39469 } } }
I20260924 11:45:42.282066 21255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:42.282382 21262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:42.287446 21262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:42.296263 21262 catalog_manager.cc:1347] Generated new cluster ID: c61966d6357d4caa859d1bfbf5d85a56
I20260924 11:45:42.296370 21262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:42.308210 21262 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:42.309983 21262 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:42.322286 21262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d35f6c50bcb4de7b83a663781da33e3: Generated new TSK 0
I20260924 11:45:42.323278 21262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:42.331578 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21177
W20260924 11:45:42.332653 21239 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38331
W20260924 11:45:42.333300 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:42.333489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21177
2026-09-24T11:45:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1779 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:42 dist-test-slave-qhrh krb5kdc[21275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:42 dist-test-slave-qhrh krb5kdc[21275](info): set up 2 sockets
Sep 24 11:45:42 dist-test-slave-qhrh krb5kdc[21275](info): commencing operation
krb5kdc: starting...
W20260924 11:45:43.368317 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:43 dist-test-slave-qhrh krb5kdc[21275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:43.583510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:43.888829 21288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:43.889092 21288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:43.889263 21288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:43.895978 21288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:43.896134 21288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:43.896257 21288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:43.896356 21288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:43.896436 21288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:43.905501 21288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:43.906648 21288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:43.907804 21288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:43.919173 21296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:43.919133 21293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:43.919203 21294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:43.919886 21288 server_base.cc:1034] running on GCE node
I20260924 11:45:43.920655 21288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:43.921907 21288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:43.923097 21288 hybrid_clock.cc:648] HybridClock initialized: now 1790250343923045 us; error 46 us; skew 500 ppm
I20260924 11:45:43.923362 21288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:43 dist-test-slave-qhrh krb5kdc[21275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:43.935508 21288 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:43.936926 21288 webserver.cc:460] Webserver started at http://127.0.0.1:36463/ using document root <none> and password file <none>
I20260924 11:45:43.937520 21288 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:43.937608 21288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:43.937860 21288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:43.940568 21288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "b4d59380a08b45408491fb818df4cc9f"
format_stamp: "Formatted at 2026-09-24 11:45:43 on dist-test-slave-qhrh"
server_key: "f682daf80fe1d18a796c5b669ab53f88"
server_key_iv: "ade7f1259f0b5b3062d61ff57570615c"
server_key_version: "encryptionkey@0"
I20260924 11:45:43.941195 21288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "b4d59380a08b45408491fb818df4cc9f"
format_stamp: "Formatted at 2026-09-24 11:45:43 on dist-test-slave-qhrh"
server_key: "f682daf80fe1d18a796c5b669ab53f88"
server_key_iv: "ade7f1259f0b5b3062d61ff57570615c"
server_key_version: "encryptionkey@0"
I20260924 11:45:43.945824 21288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:43.948741 21303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:43.949873 21288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:45:43.949988 21288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "b4d59380a08b45408491fb818df4cc9f"
format_stamp: "Formatted at 2026-09-24 11:45:43 on dist-test-slave-qhrh"
server_key: "f682daf80fe1d18a796c5b669ab53f88"
server_key_iv: "ade7f1259f0b5b3062d61ff57570615c"
server_key_version: "encryptionkey@0"
I20260924 11:45:43.950124 21288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:43.954886 21288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:43.958746 21288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:43.958974 21288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:43.975630 21288 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45607
I20260924 11:45:43.975670 21354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45607 every 8 connection(s)
I20260924 11:45:43.976881 21288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:43.980124 21355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:43.986445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21288
I20260924 11:45:43.986624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:43.987020 388 external_mini_cluster.cc:1428] Setting key dca8f0d225cbfba05346714cb09f15a2
I20260924 11:45:43.989709 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f: Bootstrap starting.
I20260924 11:45:43.993392 21355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:43.994978 21355 log.cc:824] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:43.995848 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:43.988559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47271 (local address 127.0.0.1:45607)
0924 11:45:43.989316 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:43.989330 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:43.989376 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:45:43.991575 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:43.991606 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:43.992017 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:43.992403 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:43.994268 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:43.994273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:43.994303 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:43.994317 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:43.994373 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:43.994391 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:43.994541 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:43.994688 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 11:45:43.994870 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":306,"threads_started":1}
I20260924 11:45:43.997561 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f: No bootstrap required, opened a new log
I20260924 11:45:43.998862 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:44.005796 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:44.003505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55135 (local address 127.0.0.1:45607)
0924 11:45:44.003675 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:44.003681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:44.003938 (+ 257us) server_negotiation.cc:407] Connection header received
0924 11:45:44.004113 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:44.004116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:44.004209 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:44.004322 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:44.005186 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:44.005191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:44.005199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:44.005216 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:44.005263 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:44.005280 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:44.005423 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:44.005519 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 11:45:44.005583 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 11:45:44.006356 21355 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d59380a08b45408491fb818df4cc9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45607 } }
I20260924 11:45:44.006690 21355 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:44.006788 21355 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d59380a08b45408491fb818df4cc9f, State: Initialized, Role: FOLLOWER
I20260924 11:45:44.007259 21355 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d59380a08b45408491fb818df4cc9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45607 } }
I20260924 11:45:44.007498 21355 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:44.007617 21355 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:44.007766 21355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:44.009830 21355 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d59380a08b45408491fb818df4cc9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45607 } }
I20260924 11:45:44.010272 21355 leader_election.cc:302] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d59380a08b45408491fb818df4cc9f; no voters:
I20260924 11:45:44.010804 21355 leader_election.cc:288] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:44.010914 21365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:44.012372 21365 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [term 1 LEADER]: Becoming Leader. State: Replica: b4d59380a08b45408491fb818df4cc9f, State: Running, Role: LEADER
I20260924 11:45:44.012872 21355 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:44.012817 21365 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d59380a08b45408491fb818df4cc9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45607 } }
I20260924 11:45:44.015429 21367 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d59380a08b45408491fb818df4cc9f. Latest consensus state: current_term: 1 leader_uuid: "b4d59380a08b45408491fb818df4cc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d59380a08b45408491fb818df4cc9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45607 } } }
I20260924 11:45:44.015465 21366 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d59380a08b45408491fb818df4cc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d59380a08b45408491fb818df4cc9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45607 } } }
I20260924 11:45:44.015817 21367 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:44.015872 21366 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:44.016830 21372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:44.024083 21372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:44.032635 21372 catalog_manager.cc:1347] Generated new cluster ID: 381e1845a3b24eb3a08e919d2f3ee78a
I20260924 11:45:44.032742 21372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:44.047554 21372 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:44.047971 21372 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:44.057936 21372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d59380a08b45408491fb818df4cc9f: Generated new TSK 0
I20260924 11:45:44.058825 21372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:44.063956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21288
W20260924 11:45:44.065052 21350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47271
W20260924 11:45:44.065711 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21288: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:44.065855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21288
2026-09-24T11:45:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1731 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:44 dist-test-slave-qhrh krb5kdc[21386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:44 dist-test-slave-qhrh krb5kdc[21386](info): set up 2 sockets
Sep 24 11:45:44 dist-test-slave-qhrh krb5kdc[21386](info): commencing operation
krb5kdc: starting...
W20260924 11:45:45.100339 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:45 dist-test-slave-qhrh krb5kdc[21386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:45.328953 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:45.639089 21399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:45.639395 21399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:45.639556 21399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:45.646487 21399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:45.646675 21399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:45.646787 21399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:45.646895 21399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:45.646970 21399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:45.656342 21399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:45.657564 21399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:45.658828 21399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:45.670316 21405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:45.670279 21407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:45.670364 21404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:45.671200 21399 server_base.cc:1034] running on GCE node
I20260924 11:45:45.671993 21399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:45.673337 21399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:45.674544 21399 hybrid_clock.cc:648] HybridClock initialized: now 1790250345674502 us; error 25 us; skew 500 ppm
I20260924 11:45:45.674873 21399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:45 dist-test-slave-qhrh krb5kdc[21386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:45.677759 21399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:45.679100 21399 webserver.cc:460] Webserver started at http://127.0.0.1:38863/ using document root <none> and password file <none>
I20260924 11:45:45.679744 21399 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:45.679883 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:45.680236 21399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:45.683024 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "8a2e37065e354e4bb0477d19577448b6"
format_stamp: "Formatted at 2026-09-24 11:45:45 on dist-test-slave-qhrh"
server_key: "d572eb8cc955cd446d101488b9068cbf"
server_key_iv: "4cab557c5a7af9628fe31ca3b3cb62bb"
server_key_version: "encryptionkey@0"
I20260924 11:45:45.683755 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a2e37065e354e4bb0477d19577448b6"
format_stamp: "Formatted at 2026-09-24 11:45:45 on dist-test-slave-qhrh"
server_key: "d572eb8cc955cd446d101488b9068cbf"
server_key_iv: "4cab557c5a7af9628fe31ca3b3cb62bb"
server_key_version: "encryptionkey@0"
I20260924 11:45:45.688582 21399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260924 11:45:45.691757 21414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:45.692970 21399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:45.693138 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "8a2e37065e354e4bb0477d19577448b6"
format_stamp: "Formatted at 2026-09-24 11:45:45 on dist-test-slave-qhrh"
server_key: "d572eb8cc955cd446d101488b9068cbf"
server_key_iv: "4cab557c5a7af9628fe31ca3b3cb62bb"
server_key_version: "encryptionkey@0"
I20260924 11:45:45.693318 21399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:45.706760 21399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:45.710309 21399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:45.710596 21399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:45.727691 21399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41593
I20260924 11:45:45.727728 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41593 every 8 connection(s)
I20260924 11:45:45.729043 21399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:45.730862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21399
I20260924 11:45:45.731045 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:45.731555 388 external_mini_cluster.cc:1428] Setting key ff58c1a6e37fe76e473a3ea2932ca695
I20260924 11:45:45.733290 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:45:45 dist-test-slave-qhrh krb5kdc[21386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:45:45.746317 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6: Bootstrap starting.
I20260924 11:45:45.749550 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:45.750492 21466 log.cc:824] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:45.753012 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6: No bootstrap required, opened a new log
I20260924 11:45:45.753172 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:45.733319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:59197 (local address 127.0.0.1:41593)
0924 11:45:45.734073 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:45.734084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:45.734121 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:45:45.735675 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:45.735694 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:45.736047 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:45.736440 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:45.738122 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:45.739388 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:45.741023 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:45.741657 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:45.741795 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:45.746451 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:45.746462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:45.746480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:45.746533 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:45.749446 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:45.750225 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:45.750232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:45.750237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:45.750383 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:45.750847 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:45.750856 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:45.750858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:45.751284 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:45.751654 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:45.751970 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 11:45:45.752290 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260924 11:45:45.756908 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:45.761173 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2e37065e354e4bb0477d19577448b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260924 11:45:45.761593 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:45.761744 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2e37065e354e4bb0477d19577448b6, State: Initialized, Role: FOLLOWER
I20260924 11:45:45.762378 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2e37065e354e4bb0477d19577448b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260924 11:45:45.762602 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:45.762748 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:45.762926 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:45.765735 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2e37065e354e4bb0477d19577448b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260924 11:45:45.766218 21466 leader_election.cc:302] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a2e37065e354e4bb0477d19577448b6; no voters:
I20260924 11:45:45.766773 21466 leader_election.cc:288] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:45.766868 21476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:45.768709 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:45.771544 21476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2e37065e354e4bb0477d19577448b6, State: Running, Role: LEADER
I20260924 11:45:45.772125 21476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2e37065e354e4bb0477d19577448b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260924 11:45:45.775002 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a2e37065e354e4bb0477d19577448b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2e37065e354e4bb0477d19577448b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } } }
I20260924 11:45:45.775372 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:45.775880 21478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a2e37065e354e4bb0477d19577448b6. Latest consensus state: current_term: 1 leader_uuid: "8a2e37065e354e4bb0477d19577448b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2e37065e354e4bb0477d19577448b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } } }
I20260924 11:45:45.776204 21478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:45.778378 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:45.761036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:33189 (local address 127.0.0.1:41593)
0924 11:45:45.761179 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:45.761183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:45.761566 (+ 383us) server_negotiation.cc:407] Connection header received
0924 11:45:45.761768 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:45.761773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:45.761855 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:45.761959 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:45.763642 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:45.764861 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:45.766091 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:45.766781 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:45.766903 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:45.770517 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:45.770525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:45.770535 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:45.770579 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:45.774584 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:45.775671 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:45.775677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:45.775682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:45.775878 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:45.776515 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:45.776520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:45.776523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:45.776916 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:45.777316 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:45.777659 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 11:45:45.777906 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 11:45:45.782749 21486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:45.786870 21486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:45.794255 21486 catalog_manager.cc:1347] Generated new cluster ID: 4dc94275f2654c9eb64bdadcc80fce97
I20260924 11:45:45.794373 21486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:45.806174 21486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:45.807204 21486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:45.811909 21486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a2e37065e354e4bb0477d19577448b6: Generated new TSK 0
I20260924 11:45:45.812758 21486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:45.822377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21399
I20260924 11:45:45.823395 21461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:45.910696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21399
W20260924 11:45:45.927587 21384 connection.cc:535] client connection to 127.0.0.1:41593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41593): Connection reset by peer (error 104)
2026-09-24T11:45:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:46 dist-test-slave-qhrh krb5kdc[21498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:46 dist-test-slave-qhrh krb5kdc[21498](info): set up 2 sockets
Sep 24 11:45:46 dist-test-slave-qhrh krb5kdc[21498](info): commencing operation
krb5kdc: starting...
W20260924 11:45:46.960532 388 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:47 dist-test-slave-qhrh krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:47.191586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:47.498175 21511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:47.498431 21511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:47.498628 21511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:47.505386 21511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:47.505484 21511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:47.505613 21511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:47.505723 21511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:47.505791 21511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:47.514567 21511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:47.515750 21511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:47.517050 21511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:47.528112 21519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:47.528213 21516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:47.528189 21517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:47.529153 21511 server_base.cc:1034] running on GCE node
I20260924 11:45:47.529867 21511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:47.531102 21511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:47.532320 21511 hybrid_clock.cc:648] HybridClock initialized: now 1790250347532266 us; error 43 us; skew 500 ppm
I20260924 11:45:47.532644 21511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:47 dist-test-slave-qhrh krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:47.535573 21511 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:47.536892 21511 webserver.cc:460] Webserver started at http://127.0.0.1:46679/ using document root <none> and password file <none>
I20260924 11:45:47.537487 21511 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:47.537623 21511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:47.537900 21511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:47.540668 21511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "b1430b3f445049ce812a0a53f2bffd57"
format_stamp: "Formatted at 2026-09-24 11:45:47 on dist-test-slave-qhrh"
server_key: "b90fbd1ee1ab37dd8d963ca05d950fd4"
server_key_iv: "ce93da64699d1f623d03633241a747d1"
server_key_version: "encryptionkey@0"
I20260924 11:45:47.541355 21511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "b1430b3f445049ce812a0a53f2bffd57"
format_stamp: "Formatted at 2026-09-24 11:45:47 on dist-test-slave-qhrh"
server_key: "b90fbd1ee1ab37dd8d963ca05d950fd4"
server_key_iv: "ce93da64699d1f623d03633241a747d1"
server_key_version: "encryptionkey@0"
I20260924 11:45:47.546180 21511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:47.549248 21526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:47.550369 21511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:45:47.550530 21511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "b1430b3f445049ce812a0a53f2bffd57"
format_stamp: "Formatted at 2026-09-24 11:45:47 on dist-test-slave-qhrh"
server_key: "b90fbd1ee1ab37dd8d963ca05d950fd4"
server_key_iv: "ce93da64699d1f623d03633241a747d1"
server_key_version: "encryptionkey@0"
I20260924 11:45:47.550706 21511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:47.565727 21511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:47.569818 21511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:47.570111 21511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:47.587503 21511 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46585
I20260924 11:45:47.587523 21577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46585 every 8 connection(s)
I20260924 11:45:47.588835 21511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:47.592449 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:47.593580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21511
I20260924 11:45:47.593770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:47.594208 388 external_mini_cluster.cc:1428] Setting key 93259734cb811df7a7bc168a77bf25fe
I20260924 11:45:47.603456 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57: Bootstrap starting.
I20260924 11:45:47.606918 21578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:47.607993 21578 log.cc:824] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:47.611088 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57: No bootstrap required, opened a new log
I20260924 11:45:47.611464 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:47.596241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:41735 (local address 127.0.0.1:46585)
0924 11:45:47.597117 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:47.597131 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:47.600760 (+ 3629us) server_negotiation.cc:407] Connection header received
0924 11:45:47.602569 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:47.602588 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:47.602978 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:47.603276 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:47.603871 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:47.605242 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:47.607046 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:47.607659 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:47.607777 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:47.609184 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:47.609208 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:47.609227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:47.609825 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:47.609885 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:47.609908 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:47.610012 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:47.610345 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 11:45:47.610511 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":372,"threads_started":1}
I20260924 11:45:47.614972 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:47.618897 21578 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1430b3f445049ce812a0a53f2bffd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260924 11:45:47.619357 21578 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:47.619465 21578 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1430b3f445049ce812a0a53f2bffd57, State: Initialized, Role: FOLLOWER
I20260924 11:45:47.619998 21578 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1430b3f445049ce812a0a53f2bffd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260924 11:45:47.620301 21578 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:47.620400 21578 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:47.620515 21578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:47.623117 21578 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1430b3f445049ce812a0a53f2bffd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260924 11:45:47.623675 21578 leader_election.cc:302] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1430b3f445049ce812a0a53f2bffd57; no voters:
I20260924 11:45:47.624338 21578 leader_election.cc:288] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:47.624466 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:47.626405 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [term 1 LEADER]: Becoming Leader. State: Replica: b1430b3f445049ce812a0a53f2bffd57, State: Running, Role: LEADER
I20260924 11:45:47.626711 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:47.618976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:59737 (local address 127.0.0.1:46585)
0924 11:45:47.619113 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:47.619116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:47.619655 (+ 539us) server_negotiation.cc:407] Connection header received
0924 11:45:47.619862 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:47.619866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:47.619921 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:47.620020 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:47.620658 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:47.621851 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:47.623161 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:47.623848 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:47.623962 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:47.625330 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:47.625360 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:47.625370 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:47.625391 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:47.625488 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:47.625521 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:47.626143 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:47.626239 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 11:45:47.626302 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 11:45:47.626948 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1430b3f445049ce812a0a53f2bffd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260924 11:45:47.627321 21578 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:47.630087 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1430b3f445049ce812a0a53f2bffd57. Latest consensus state: current_term: 1 leader_uuid: "b1430b3f445049ce812a0a53f2bffd57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1430b3f445049ce812a0a53f2bffd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } } }
I20260924 11:45:47.630204 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1430b3f445049ce812a0a53f2bffd57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1430b3f445049ce812a0a53f2bffd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } } }
I20260924 11:45:47.630512 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:47.631309 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:47.637861 21593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:47.642733 21593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:47.651134 21593 catalog_manager.cc:1347] Generated new cluster ID: 62a6f947617c47e8b9a1714691bf7cb2
I20260924 11:45:47.651254 21593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:47.668030 21593 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:47.669157 21593 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:47.678362 21593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1430b3f445049ce812a0a53f2bffd57: Generated new TSK 0
I20260924 11:45:47.679003 21593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:47.686550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21511
W20260924 11:45:47.687604 21573 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:41735
W20260924 11:45:47.688272 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21511: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:47.688410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21511
W20260924 11:45:47.704766 21496 connection.cc:535] client connection to 127.0.0.1:46585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46585): Connection reset by peer (error 104)
2026-09-24T11:45:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:47 dist-test-slave-qhrh krb5kdc[21609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:47 dist-test-slave-qhrh krb5kdc[21609](info): set up 2 sockets
Sep 24 11:45:47 dist-test-slave-qhrh krb5kdc[21609](info): commencing operation
krb5kdc: starting...
W20260924 11:45:48.724490 388 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:48 dist-test-slave-qhrh krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:48.946612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:49.258747 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:49.259006 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:49.259246 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:49.266362 21622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:49.266489 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:49.266574 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:49.266672 21622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:49.266741 21622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:49.275695 21622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:49.276880 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:49.278167 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:49.289358 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:49.289321 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:49.289435 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:49.289693 21622 server_base.cc:1034] running on GCE node
I20260924 11:45:49.290583 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:49.291818 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:49.293036 21622 hybrid_clock.cc:648] HybridClock initialized: now 1790250349293004 us; error 50 us; skew 500 ppm
I20260924 11:45:49.293356 21622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:49 dist-test-slave-qhrh krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:49.306411 21622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:49.307957 21622 webserver.cc:460] Webserver started at http://127.0.0.1:46429/ using document root <none> and password file <none>
I20260924 11:45:49.308696 21622 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:49.308832 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:49.309154 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:49.311936 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "e2a33699a1db4b579a7c70f7a7353e4d"
format_stamp: "Formatted at 2026-09-24 11:45:49 on dist-test-slave-qhrh"
server_key: "1027ee6329eeeab06034dcd3e31c3ca8"
server_key_iv: "3ed94d97a5168c29932094ab88de910b"
server_key_version: "encryptionkey@0"
I20260924 11:45:49.312753 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "e2a33699a1db4b579a7c70f7a7353e4d"
format_stamp: "Formatted at 2026-09-24 11:45:49 on dist-test-slave-qhrh"
server_key: "1027ee6329eeeab06034dcd3e31c3ca8"
server_key_iv: "3ed94d97a5168c29932094ab88de910b"
server_key_version: "encryptionkey@0"
I20260924 11:45:49.317641 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:45:49.320845 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:49.322130 21622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:45:49.322294 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "e2a33699a1db4b579a7c70f7a7353e4d"
format_stamp: "Formatted at 2026-09-24 11:45:49 on dist-test-slave-qhrh"
server_key: "1027ee6329eeeab06034dcd3e31c3ca8"
server_key_iv: "3ed94d97a5168c29932094ab88de910b"
server_key_version: "encryptionkey@0"
I20260924 11:45:49.322484 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:49.328063 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:49.331667 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:49.331949 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:49.349059 21622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40751
I20260924 11:45:49.349102 21688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40751 every 8 connection(s)
I20260924 11:45:49.350275 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:49.353492 21689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:49.357447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21622
I20260924 11:45:49.357606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:49.358094 388 external_mini_cluster.cc:1428] Setting key 3a0dc44903c4c09a4a1ef6f9c9361682
I20260924 11:45:49.364271 21689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d: Bootstrap starting.
I20260924 11:45:49.366169 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:49.359708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:56157 (local address 127.0.0.1:40751)
0924 11:45:49.361034 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:49.361046 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:49.361088 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:45:49.362970 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:49.362990 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:49.363285 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:49.363682 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:49.364688 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:49.364694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:49.364705 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:49.364721 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:49.364772 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:49.364790 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:49.365012 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:49.365145 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 11:45:49.365321 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":954,"threads_started":1}
I20260924 11:45:49.367544 21689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:49.368691 21689 log.cc:824] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:49.369697 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:49.371409 21689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d: No bootstrap required, opened a new log
I20260924 11:45:49.376530 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:49.374163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:34705 (local address 127.0.0.1:40751)
0924 11:45:49.374334 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:49.374356 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:49.374637 (+ 281us) server_negotiation.cc:407] Connection header received
0924 11:45:49.374795 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:49.374799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:49.374845 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:49.374943 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:49.375824 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:49.375828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:49.375831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:49.375844 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:49.375881 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:49.375897 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:49.375986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:49.376116 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 11:45:49.376217 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:45:49.379993 21689 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 11:45:49.380374 21689 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:49.380479 21689 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a33699a1db4b579a7c70f7a7353e4d, State: Initialized, Role: FOLLOWER
I20260924 11:45:49.381037 21689 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 11:45:49.381283 21689 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:49.381428 21689 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:49.381593 21689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:49.383709 21689 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 11:45:49.384241 21689 leader_election.cc:302] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a33699a1db4b579a7c70f7a7353e4d; no voters:
I20260924 11:45:49.384832 21689 leader_election.cc:288] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:49.385010 21699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:49.385638 21699 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [term 1 LEADER]: Becoming Leader. State: Replica: e2a33699a1db4b579a7c70f7a7353e4d, State: Running, Role: LEADER
I20260924 11:45:49.386118 21699 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 11:45:49.386756 21689 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:49.388635 21700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260924 11:45:49.389079 21700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:49.393169 21707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:49.393103 21701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a33699a1db4b579a7c70f7a7353e4d. Latest consensus state: current_term: 1 leader_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a33699a1db4b579a7c70f7a7353e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260924 11:45:49.393452 21701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:49.397881 21707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:49.405769 21707 catalog_manager.cc:1347] Generated new cluster ID: 8cf1171a94814af3a2db5d6f5db3bd1d
I20260924 11:45:49.405887 21707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:49.423632 21707 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:49.423986 21707 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:49.436949 21707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2a33699a1db4b579a7c70f7a7353e4d: Generated new TSK 0
I20260924 11:45:49.437652 21707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:49.447893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21622
W20260924 11:45:49.448868 21684 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:56157
W20260924 11:45:49.449455 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21622: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:49.449563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21622
2026-09-24T11:45:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1761 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:49 dist-test-slave-qhrh krb5kdc[21720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:49 dist-test-slave-qhrh krb5kdc[21720](info): set up 2 sockets
Sep 24 11:45:49 dist-test-slave-qhrh krb5kdc[21720](info): commencing operation
krb5kdc: starting...
W20260924 11:45:50.508442 388 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:50 dist-test-slave-qhrh krb5kdc[21720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:50.726925 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:51.034682 21733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:51.034952 21733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:51.035039 21733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:51.041877 21733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:51.041982 21733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:51.042026 21733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:51.042085 21733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:51.042130 21733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:51.051029 21733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:51.052184 21733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:51.053354 21733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:51.064597 21741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:51.064723 21738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:51.064693 21739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:51.065977 21733 server_base.cc:1034] running on GCE node
I20260924 11:45:51.066694 21733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:51.067963 21733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:51.069180 21733 hybrid_clock.cc:648] HybridClock initialized: now 1790250351069145 us; error 43 us; skew 500 ppm
I20260924 11:45:51.069504 21733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:51 dist-test-slave-qhrh krb5kdc[21720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:51.072413 21733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:51.073923 21733 webserver.cc:460] Webserver started at http://127.0.0.1:46751/ using document root <none> and password file <none>
I20260924 11:45:51.074537 21733 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:51.074678 21733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:51.074990 21733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:51.077901 21733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "c023da9203fd4c9e867993c00cacdaa4"
format_stamp: "Formatted at 2026-09-24 11:45:51 on dist-test-slave-qhrh"
server_key: "0db43bef1278efe4361c4895f5e856a3"
server_key_iv: "6ee50cc77d3b0e420e1777745d8e3924"
server_key_version: "encryptionkey@0"
I20260924 11:45:51.078626 21733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "c023da9203fd4c9e867993c00cacdaa4"
format_stamp: "Formatted at 2026-09-24 11:45:51 on dist-test-slave-qhrh"
server_key: "0db43bef1278efe4361c4895f5e856a3"
server_key_iv: "6ee50cc77d3b0e420e1777745d8e3924"
server_key_version: "encryptionkey@0"
I20260924 11:45:51.083398 21733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:45:51.086537 21748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:51.087725 21733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:45:51.087903 21733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "c023da9203fd4c9e867993c00cacdaa4"
format_stamp: "Formatted at 2026-09-24 11:45:51 on dist-test-slave-qhrh"
server_key: "0db43bef1278efe4361c4895f5e856a3"
server_key_iv: "6ee50cc77d3b0e420e1777745d8e3924"
server_key_version: "encryptionkey@0"
I20260924 11:45:51.088104 21733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:51.103497 21733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:51.107339 21733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:51.107664 21733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:51.125687 21733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44235
I20260924 11:45:51.125705 21799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44235 every 8 connection(s)
I20260924 11:45:51.127077 21733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:51.130479 21800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:51.137337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21733
I20260924 11:45:51.137517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:51.137887 388 external_mini_cluster.cc:1428] Setting key 279e11c53852c5ce1c3662bfdfc27c89
I20260924 11:45:51.140048 21800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4: Bootstrap starting.
I20260924 11:45:51.143173 21800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:51.144414 21800 log.cc:824] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:51.147362 21800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4: No bootstrap required, opened a new log
Sep 24 11:45:51 dist-test-slave-qhrh krb5kdc[21720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:45:51.155889 21800 raft_consensus.cc:348] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c023da9203fd4c9e867993c00cacdaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44235 } }
I20260924 11:45:51.156255 21800 raft_consensus.cc:374] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:51.156366 21800 raft_consensus.cc:729] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c023da9203fd4c9e867993c00cacdaa4, State: Initialized, Role: FOLLOWER
I20260924 11:45:51.157006 21800 consensus_queue.cc:260] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c023da9203fd4c9e867993c00cacdaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44235 } }
I20260924 11:45:51.157303 21800 raft_consensus.cc:388] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:51.157464 21800 raft_consensus.cc:482] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:51.157655 21800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:51.159966 21800 raft_consensus.cc:504] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c023da9203fd4c9e867993c00cacdaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44235 } }
I20260924 11:45:51.160236 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:51.139548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:52323 (local address 127.0.0.1:44235)
0924 11:45:51.140324 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:51.140334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:51.140371 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:45:51.142093 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:51.142113 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:51.142469 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:51.142762 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:51.144473 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:51.146137 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:51.147812 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:51.148479 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:51.148612 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:51.153029 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:51.153035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:51.153047 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:51.153079 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:51.155905 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:51.156779 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:51.156786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:51.156792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:51.157025 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:51.157557 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:51.157563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:51.157566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:51.158130 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:51.158494 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:51.158971 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 11:45:51.159286 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":354,"threads_started":1}
I20260924 11:45:51.160521 21800 leader_election.cc:302] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c023da9203fd4c9e867993c00cacdaa4; no voters:
I20260924 11:45:51.161170 21800 leader_election.cc:288] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:51.161281 21805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:51.162348 21805 raft_consensus.cc:686] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [term 1 LEADER]: Becoming Leader. State: Replica: c023da9203fd4c9e867993c00cacdaa4, State: Running, Role: LEADER
I20260924 11:45:51.162930 21805 consensus_queue.cc:237] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c023da9203fd4c9e867993c00cacdaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44235 } }
I20260924 11:45:51.163419 21800 sys_catalog.cc:564] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:51.164075 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:51.165913 21807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c023da9203fd4c9e867993c00cacdaa4. Latest consensus state: current_term: 1 leader_uuid: "c023da9203fd4c9e867993c00cacdaa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c023da9203fd4c9e867993c00cacdaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44235 } } }
I20260924 11:45:51.166366 21807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:51.167240 21810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:51.166992 21806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c023da9203fd4c9e867993c00cacdaa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c023da9203fd4c9e867993c00cacdaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44235 } } }
I20260924 11:45:51.167517 21806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:51.183766 21810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:51.193045 21810 catalog_manager.cc:1347] Generated new cluster ID: c1721851ea284c5ea97b2c51163bd163
I20260924 11:45:51.193195 21810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:51.201970 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:51.186451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:56601 (local address 127.0.0.1:44235)
0924 11:45:51.186623 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:51.186629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:51.187146 (+ 517us) server_negotiation.cc:407] Connection header received
0924 11:45:51.187324 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:51.187334 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:51.187420 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:51.187534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:51.189359 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:51.190288 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:51.191845 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:51.192437 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:51.192563 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:51.195596 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:51.195603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:51.195607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:51.195643 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:51.199072 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:51.199881 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:51.199886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:51.199890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:51.200032 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:51.200462 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:51.200466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:51.200470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:51.200819 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:51.201078 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:51.201397 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 11:45:51.201575 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:45:51.223600 21810 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:51.224906 21810 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:51.233856 21810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c023da9203fd4c9e867993c00cacdaa4: Generated new TSK 0
I20260924 11:45:51.234844 21810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:51.239935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21733
I20260924 11:45:51.241343 21795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:51.329185 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21733
W20260924 11:45:51.345676 21718 connection.cc:535] client connection to 127.0.0.1:44235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44235): Connection reset by peer (error 104)
2026-09-24T11:45:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:51 dist-test-slave-qhrh krb5kdc[21832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:51 dist-test-slave-qhrh krb5kdc[21832](info): set up 2 sockets
Sep 24 11:45:51 dist-test-slave-qhrh krb5kdc[21832](info): commencing operation
krb5kdc: starting...
W20260924 11:45:52.380679 388 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.011s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:52 dist-test-slave-qhrh krb5kdc[21832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:52.606997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:52.917238 21845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:52.917560 21845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:52.917752 21845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:52.924814 21845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:52.924949 21845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:52.925051 21845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:52.925172 21845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:52.925264 21845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:52.934302 21845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:52.935509 21845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:52.936880 21845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:52.947902 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:52.947862 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:52.947961 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:52.953665 21845 server_base.cc:1034] running on GCE node
I20260924 11:45:52.954367 21845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:52.955679 21845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:52.956872 21845 hybrid_clock.cc:648] HybridClock initialized: now 1790250352956817 us; error 47 us; skew 500 ppm
I20260924 11:45:52.957199 21845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:52 dist-test-slave-qhrh krb5kdc[21832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:52.960089 21845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:52.961457 21845 webserver.cc:460] Webserver started at http://127.0.0.1:37671/ using document root <none> and password file <none>
I20260924 11:45:52.962088 21845 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:52.962193 21845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:52.962442 21845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:52.965234 21845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f"
format_stamp: "Formatted at 2026-09-24 11:45:52 on dist-test-slave-qhrh"
server_key: "85a205413679955e053fd29adf6dff69"
server_key_iv: "b6b576c1741179c61bca8cd85b2c06e9"
server_key_version: "encryptionkey@0"
I20260924 11:45:52.965924 21845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f"
format_stamp: "Formatted at 2026-09-24 11:45:52 on dist-test-slave-qhrh"
server_key: "85a205413679955e053fd29adf6dff69"
server_key_iv: "b6b576c1741179c61bca8cd85b2c06e9"
server_key_version: "encryptionkey@0"
I20260924 11:45:52.970701 21845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:52.973908 21860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:52.975122 21845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:45:52.975332 21845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f"
format_stamp: "Formatted at 2026-09-24 11:45:52 on dist-test-slave-qhrh"
server_key: "85a205413679955e053fd29adf6dff69"
server_key_iv: "b6b576c1741179c61bca8cd85b2c06e9"
server_key_version: "encryptionkey@0"
I20260924 11:45:52.975518 21845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:52.993158 21845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:52.996829 21845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:52.997110 21845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:53.016439 21845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38441
I20260924 11:45:53.016439 21911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38441 every 8 connection(s)
I20260924 11:45:53.018124 21845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:53.021816 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:53.030678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21845
I20260924 11:45:53.030869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:53.031399 388 external_mini_cluster.cc:1428] Setting key af882f6b1c53bf742f15f8b0f547d543
I20260924 11:45:53.036228 21912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f: Bootstrap starting.
I20260924 11:45:53.039988 21912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:53.041033 21912 log.cc:824] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:53.043870 21912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f: No bootstrap required, opened a new log
I20260924 11:45:53.045811 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:53.032954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:43939 (local address 127.0.0.1:38441)
0924 11:45:53.033683 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:53.033701 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:53.033750 (+ 49us) server_negotiation.cc:407] Connection header received
0924 11:45:53.036051 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:53.036076 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:53.036622 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:53.037052 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:53.037529 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:53.039509 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:53.040830 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:53.041808 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:53.041955 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:53.043281 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:53.043290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:53.043306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:53.043894 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:53.043975 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:53.044008 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:53.044165 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:53.044469 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 11:45:53.044615 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":329,"threads_started":1}
I20260924 11:45:53.049374 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:53.051959 21912 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38441 } }
I20260924 11:45:53.052338 21912 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:53.052448 21912 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d1406b98af4dc0b5b7ae6da3c0209f, State: Initialized, Role: FOLLOWER
I20260924 11:45:53.053033 21912 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38441 } }
I20260924 11:45:53.053333 21912 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:53.053501 21912 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:53.053680 21912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:53.056653 21912 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38441 } }
I20260924 11:45:53.057139 21912 leader_election.cc:302] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1d1406b98af4dc0b5b7ae6da3c0209f; no voters:
I20260924 11:45:53.057749 21912 leader_election.cc:288] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:53.057842 21922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:53.058946 21922 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [term 1 LEADER]: Becoming Leader. State: Replica: e1d1406b98af4dc0b5b7ae6da3c0209f, State: Running, Role: LEADER
I20260924 11:45:53.059566 21922 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38441 } }
I20260924 11:45:53.060209 21912 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:53.061107 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:53.053335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:39537 (local address 127.0.0.1:38441)
0924 11:45:53.053491 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:53.053495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:53.053984 (+ 489us) server_negotiation.cc:407] Connection header received
0924 11:45:53.054114 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:53.054122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:53.054208 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:53.054312 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:53.054714 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:53.055853 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:53.057190 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:53.057920 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:53.058075 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:53.059935 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:53.059940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:53.059945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:53.059964 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:53.060034 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:53.060052 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:53.060146 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:53.060705 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 11:45:53.060781 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:45:53.061846 21924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1d1406b98af4dc0b5b7ae6da3c0209f. Latest consensus state: current_term: 1 leader_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38441 } } }
I20260924 11:45:53.062180 21924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:53.062892 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d1406b98af4dc0b5b7ae6da3c0209f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38441 } } }
I20260924 11:45:53.063199 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:53.064469 21931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:53.070173 21931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:53.078474 21931 catalog_manager.cc:1347] Generated new cluster ID: e84ea30df150424d962f014f52d356a8
I20260924 11:45:53.078563 21931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:53.094520 21931 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:53.096194 21931 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:53.103128 21931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1d1406b98af4dc0b5b7ae6da3c0209f: Generated new TSK 0
I20260924 11:45:53.103999 21931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:53.118284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21845
W20260924 11:45:53.119688 21907 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:43939
W20260924 11:45:53.120420 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21845: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:53.120565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21845
W20260924 11:45:53.136695 21830 connection.cc:535] client connection to 127.0.0.1:38441 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38441): Connection reset by peer (error 104)
2026-09-24T11:45:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1793 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:53 dist-test-slave-qhrh krb5kdc[21943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:53 dist-test-slave-qhrh krb5kdc[21943](info): set up 2 sockets
Sep 24 11:45:53 dist-test-slave-qhrh krb5kdc[21943](info): commencing operation
krb5kdc: starting...
W20260924 11:45:54.168648 388 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:54 dist-test-slave-qhrh krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:45:54.392861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:54.693739 21956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:54.694025 21956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:54.694167 21956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:54.700680 21956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:54.700814 21956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:54.700915 21956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:54.701016 21956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:54.701099 21956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:54.709869 21956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:54.711081 21956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:54.712352 21956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:54.723536 21964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:54.723599 21961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:54.723575 21962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:54.724828 21956 server_base.cc:1034] running on GCE node
I20260924 11:45:54.725579 21956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:54.726868 21956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:54.728086 21956 hybrid_clock.cc:648] HybridClock initialized: now 1790250354728055 us; error 53 us; skew 500 ppm
I20260924 11:45:54.728421 21956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:54 dist-test-slave-qhrh krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:54.741888 21956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:45:54.743328 21956 webserver.cc:460] Webserver started at http://127.0.0.1:46189/ using document root <none> and password file <none>
I20260924 11:45:54.743953 21956 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:54.744091 21956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:54.744411 21956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:54.747226 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "d4f0b3d1fd21400eb4232723a285e6c5"
format_stamp: "Formatted at 2026-09-24 11:45:54 on dist-test-slave-qhrh"
server_key: "8e47313ad2371a3fd6b0fe16ea27c26e"
server_key_iv: "2b9c63219ab1ecf28f5673fc7037d948"
server_key_version: "encryptionkey@0"
I20260924 11:45:54.747991 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "d4f0b3d1fd21400eb4232723a285e6c5"
format_stamp: "Formatted at 2026-09-24 11:45:54 on dist-test-slave-qhrh"
server_key: "8e47313ad2371a3fd6b0fe16ea27c26e"
server_key_iv: "2b9c63219ab1ecf28f5673fc7037d948"
server_key_version: "encryptionkey@0"
I20260924 11:45:54.752869 21956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:45:54.756050 21971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:54.757265 21956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:45:54.757433 21956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "d4f0b3d1fd21400eb4232723a285e6c5"
format_stamp: "Formatted at 2026-09-24 11:45:54 on dist-test-slave-qhrh"
server_key: "8e47313ad2371a3fd6b0fe16ea27c26e"
server_key_iv: "2b9c63219ab1ecf28f5673fc7037d948"
server_key_version: "encryptionkey@0"
I20260924 11:45:54.757612 21956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:54.763011 21956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:54.767019 21956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:54.767298 21956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:54.784101 21956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45503
I20260924 11:45:54.784179 22022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45503 every 8 connection(s)
I20260924 11:45:54.785310 21956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:54.788781 22023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:54.792678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 21956
I20260924 11:45:54.792855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:54.793236 388 external_mini_cluster.cc:1428] Setting key a46d1b10f81d3015fc9ad43cc00de844
I20260924 11:45:54.799167 22023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5: Bootstrap starting.
I20260924 11:45:54.800827 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:54.794755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:39181 (local address 127.0.0.1:45503)
0924 11:45:54.795430 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:54.795441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:54.795494 (+ 53us) server_negotiation.cc:407] Connection header received
0924 11:45:54.797779 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:54.797799 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:54.798104 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:54.798375 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:54.799343 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:54.799348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:54.799359 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:54.799372 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:54.799429 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:54.799465 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:54.799614 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:54.799768 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 11:45:54.799947 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":304,"threads_started":1}
I20260924 11:45:54.802275 22023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:54.803426 22023 log.cc:824] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:54.804277 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:54.806310 22023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5: No bootstrap required, opened a new log
I20260924 11:45:54.811504 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:54.809132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:46425 (local address 127.0.0.1:45503)
0924 11:45:54.809321 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:54.809330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:54.809828 (+ 498us) server_negotiation.cc:407] Connection header received
0924 11:45:54.809920 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:54.809926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:54.809990 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:54.810095 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:54.810878 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:54.810884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:54.810888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:54.810905 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:54.810948 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:54.810971 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:54.811074 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:54.811171 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 11:45:54.811226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 11:45:54.814448 22023 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260924 11:45:54.814760 22023 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:54.814885 22023 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f0b3d1fd21400eb4232723a285e6c5, State: Initialized, Role: FOLLOWER
I20260924 11:45:54.815448 22023 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260924 11:45:54.815694 22023 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:54.815848 22023 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:54.816027 22023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:54.818002 22023 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260924 11:45:54.818495 22023 leader_election.cc:302] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f0b3d1fd21400eb4232723a285e6c5; no voters:
I20260924 11:45:54.819020 22023 leader_election.cc:288] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:54.819113 22033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:54.819741 22033 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [term 1 LEADER]: Becoming Leader. State: Replica: d4f0b3d1fd21400eb4232723a285e6c5, State: Running, Role: LEADER
I20260924 11:45:54.820211 22033 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260924 11:45:54.820917 22023 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:54.823261 22035 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f0b3d1fd21400eb4232723a285e6c5. Latest consensus state: current_term: 1 leader_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } } }
I20260924 11:45:54.823236 22034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f0b3d1fd21400eb4232723a285e6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } } }
I20260924 11:45:54.823673 22035 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:54.823700 22034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:54.829335 22038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:54.835193 22038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:54.843272 22038 catalog_manager.cc:1347] Generated new cluster ID: e6b9b146453646fba3a109e0bbeb2089
I20260924 11:45:54.843382 22038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:54.853402 22038 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:54.853761 22038 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:54.860419 22038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f0b3d1fd21400eb4232723a285e6c5: Generated new TSK 0
I20260924 11:45:54.861052 22038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:54.870236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21956
W20260924 11:45:54.871215 22018 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:39181
W20260924 11:45:54.871809 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 21956: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:54.871948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 21956
2026-09-24T11:45:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1747 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:54 dist-test-slave-qhrh krb5kdc[22054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:54 dist-test-slave-qhrh krb5kdc[22054](info): set up 2 sockets
Sep 24 11:45:54 dist-test-slave-qhrh krb5kdc[22054](info): commencing operation
krb5kdc: starting...
W20260924 11:45:55.924541 388 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:56 dist-test-slave-qhrh krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:45:56.141088 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40945
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:56.442669 22067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:56.442927 22067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:56.443109 22067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:56.449743 22067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:56.449862 22067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:56.449957 22067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:56.450062 22067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:56.450119 22067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:56.458912 22067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:56.460120 22067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:56.461403 22067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:56.472632 22073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:56.472661 22072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:56.472909 22075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:56.473340 22067 server_base.cc:1034] running on GCE node
I20260924 11:45:56.474035 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:56.475378 22067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:56.476591 22067 hybrid_clock.cc:648] HybridClock initialized: now 1790250356476548 us; error 48 us; skew 500 ppm
I20260924 11:45:56.476917 22067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:56 dist-test-slave-qhrh krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:56.479732 22067 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:45:56.481005 22067 webserver.cc:460] Webserver started at http://127.0.97.62:39647/ using document root <none> and password file <none>
I20260924 11:45:56.481590 22067 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:56.481715 22067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:56.482012 22067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:56.484793 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "148cb376774c44d49ff2bd08f86c2a05"
format_stamp: "Formatted at 2026-09-24 11:45:56 on dist-test-slave-qhrh"
server_key: "dc37eeb389d0931ddf71ae47a823186f"
server_key_iv: "3e4381e3bd429632938bfc652c3928e8"
server_key_version: "encryptionkey@0"
I20260924 11:45:56.485459 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "148cb376774c44d49ff2bd08f86c2a05"
format_stamp: "Formatted at 2026-09-24 11:45:56 on dist-test-slave-qhrh"
server_key: "dc37eeb389d0931ddf71ae47a823186f"
server_key_iv: "3e4381e3bd429632938bfc652c3928e8"
server_key_version: "encryptionkey@0"
I20260924 11:45:56.490140 22067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:45:56.493243 22082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:56.494372 22067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:45:56.494526 22067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "148cb376774c44d49ff2bd08f86c2a05"
format_stamp: "Formatted at 2026-09-24 11:45:56 on dist-test-slave-qhrh"
server_key: "dc37eeb389d0931ddf71ae47a823186f"
server_key_iv: "3e4381e3bd429632938bfc652c3928e8"
server_key_version: "encryptionkey@0"
I20260924 11:45:56.494699 22067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:56.526925 22067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:56.530510 22067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:56.530774 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:56.547904 22067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40945
I20260924 11:45:56.547941 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40945 every 8 connection(s)
I20260924 11:45:56.549264 22067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:56.550860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22067
I20260924 11:45:56.551029 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:56.551476 388 external_mini_cluster.cc:1428] Setting key f61dc499a3fab937f55b846d82093245
I20260924 11:45:56.553224 22134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:45:56 dist-test-slave-qhrh krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:45:56.566047 22134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05: Bootstrap starting.
I20260924 11:45:56.569026 22134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:56.570037 22134 log.cc:824] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:56.572643 22134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05: No bootstrap required, opened a new log
I20260924 11:45:56.573411 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:56.553091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58505 (local address 127.0.97.62:40945)
0924 11:45:56.553911 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:56.553921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:56.553959 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:45:56.555413 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:56.555432 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:56.555832 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:56.556180 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:56.557822 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:56.559077 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:56.560831 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:56.561427 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:56.561552 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:56.566184 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:56.566195 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:56.566213 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:56.566259 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:56.569303 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:56.570100 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:56.570106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:56.570112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:56.570315 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:56.570767 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:56.570771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:56.570774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:56.571227 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:56.571592 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:56.572039 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 11:45:56.572394 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":411,"threads_started":1}
I20260924 11:45:56.576984 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:56.580021 22134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148cb376774c44d49ff2bd08f86c2a05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40945 } }
I20260924 11:45:56.580401 22134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:56.580494 22134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148cb376774c44d49ff2bd08f86c2a05, State: Initialized, Role: FOLLOWER
I20260924 11:45:56.581183 22134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148cb376774c44d49ff2bd08f86c2a05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40945 } }
I20260924 11:45:56.581423 22134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:56.581544 22134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:56.581715 22134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:56.583751 22134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148cb376774c44d49ff2bd08f86c2a05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40945 } }
I20260924 11:45:56.584223 22134 leader_election.cc:302] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 148cb376774c44d49ff2bd08f86c2a05; no voters:
I20260924 11:45:56.584846 22134 leader_election.cc:288] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:56.584923 22144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:56.586133 22144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [term 1 LEADER]: Becoming Leader. State: Replica: 148cb376774c44d49ff2bd08f86c2a05, State: Running, Role: LEADER
I20260924 11:45:56.586746 22144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148cb376774c44d49ff2bd08f86c2a05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40945 } }
I20260924 11:45:56.587185 22134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:56.589450 22146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 148cb376774c44d49ff2bd08f86c2a05. Latest consensus state: current_term: 1 leader_uuid: "148cb376774c44d49ff2bd08f86c2a05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148cb376774c44d49ff2bd08f86c2a05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40945 } } }
I20260924 11:45:56.589897 22146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:56.590512 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "148cb376774c44d49ff2bd08f86c2a05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148cb376774c44d49ff2bd08f86c2a05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40945 } } }
I20260924 11:45:56.590834 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05 [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:56.598280 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:56.580923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52649 (local address 127.0.97.62:40945)
0924 11:45:56.581083 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:56.581088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:56.581548 (+ 460us) server_negotiation.cc:407] Connection header received
0924 11:45:56.581691 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:56.581695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:56.581754 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:56.581839 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:56.583334 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:56.584044 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:56.585432 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:56.586278 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:56.586399 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:56.589371 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:56.589378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:56.589384 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:45:56.589427 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:56.593769 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:56.594863 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:56.594871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:56.594877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:56.595091 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:45:56.595767 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:45:56.595775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:45:56.595792 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:45:56.596281 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:56.596627 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:56.597502 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 11:45:56.597736 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:45:56.600766 22149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:56.605594 22149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:56.614130 22149 catalog_manager.cc:1347] Generated new cluster ID: 8c7853fd6a7542acb06f01639b97cc38
I20260924 11:45:56.614224 22149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:56.622680 22149 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:56.624280 22149 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:56.641882 22149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 148cb376774c44d49ff2bd08f86c2a05: Generated new TSK 0
I20260924 11:45:56.642902 22149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:56.655355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22067
I20260924 11:45:56.656702 22129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:45:56.741148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22067
W20260924 11:45:56.757296 22052 connection.cc:535] client connection to 127.0.97.62:40945 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40945): Connection reset by peer (error 104)
2026-09-24T11:45:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:56 dist-test-slave-qhrh krb5kdc[22167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:56 dist-test-slave-qhrh krb5kdc[22167](info): set up 2 sockets
Sep 24 11:45:56 dist-test-slave-qhrh krb5kdc[22167](info): commencing operation
krb5kdc: starting...
W20260924 11:45:57.772406 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:57 dist-test-slave-qhrh krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:45:57.986537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40723
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:45:58.290621 22180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:45:58.290906 22180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:45:58.291106 22180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:45:58.297874 22180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:45:58.298010 22180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:45:58.298134 22180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:45:58.298233 22180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:45:58.298296 22180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:45:58.307531 22180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:45:58.308818 22180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:45:58.310142 22180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:45:58.321906 22185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:58.321849 22186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:45:58.321941 22188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:45:58.322826 22180 server_base.cc:1034] running on GCE node
I20260924 11:45:58.323587 22180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:45:58.324887 22180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:45:58.326099 22180 hybrid_clock.cc:648] HybridClock initialized: now 1790250358326056 us; error 53 us; skew 500 ppm
I20260924 11:45:58.326432 22180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:45:58 dist-test-slave-qhrh krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:45:58.329433 22180 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:45:58.330724 22180 webserver.cc:460] Webserver started at http://127.0.97.62:33003/ using document root <none> and password file <none>
I20260924 11:45:58.331373 22180 fs_manager.cc:359] Metadata directory not provided
I20260924 11:45:58.331513 22180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:45:58.331807 22180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:45:58.334605 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "c18ae7d0b9434fc49cd6396210c2911a"
format_stamp: "Formatted at 2026-09-24 11:45:58 on dist-test-slave-qhrh"
server_key: "c37c620d4510dd2cd076cf522521e9a2"
server_key_iv: "9ebf8a8bd7b82f726497618a79439cb1"
server_key_version: "encryptionkey@0"
I20260924 11:45:58.335327 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "c18ae7d0b9434fc49cd6396210c2911a"
format_stamp: "Formatted at 2026-09-24 11:45:58 on dist-test-slave-qhrh"
server_key: "c37c620d4510dd2cd076cf522521e9a2"
server_key_iv: "9ebf8a8bd7b82f726497618a79439cb1"
server_key_version: "encryptionkey@0"
I20260924 11:45:58.340106 22180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 11:45:58.343205 22195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:45:58.344399 22180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:45:58.344552 22180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "c18ae7d0b9434fc49cd6396210c2911a"
format_stamp: "Formatted at 2026-09-24 11:45:58 on dist-test-slave-qhrh"
server_key: "c37c620d4510dd2cd076cf522521e9a2"
server_key_iv: "9ebf8a8bd7b82f726497618a79439cb1"
server_key_version: "encryptionkey@0"
I20260924 11:45:58.344722 22180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:45:58.362918 22180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:45:58.366513 22180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:45:58.366776 22180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:45:58.383659 22180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40723
I20260924 11:45:58.383689 22246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40723 every 8 connection(s)
I20260924 11:45:58.384842 22180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:45:58.386721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22180
I20260924 11:45:58.386893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:45:58.387357 388 external_mini_cluster.cc:1428] Setting key e95648276f3af706fa5ce5780f0bc388
I20260924 11:45:58.388859 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:45:58.401274 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:58.389074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42559 (local address 127.0.97.62:40723)
0924 11:45:58.389837 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:58.389851 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:58.389896 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:45:58.391847 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:58.391883 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:58.392363 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:58.392830 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:58.393305 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:58.395231 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:58.396580 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:58.397521 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:58.397683 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:58.398718 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:58.398732 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:58.398747 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:58.399405 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:58.399502 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:58.399546 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:58.399703 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:58.399915 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 11:45:58.400074 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260924 11:45:58.402587 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a: Bootstrap starting.
I20260924 11:45:58.404733 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:45:58.405682 22247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a: Neither blocks nor log segments found. Creating new log.
I20260924 11:45:58.406819 22247 log.cc:824] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:45:58.409929 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a: No bootstrap required, opened a new log
I20260924 11:45:58.415851 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:45:58.409627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60589 (local address 127.0.97.62:40723)
0924 11:45:58.409829 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:45:58.409835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:45:58.410238 (+ 403us) server_negotiation.cc:407] Connection header received
0924 11:45:58.410366 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:45:58.410373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:45:58.410461 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:45:58.410571 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:45:58.411000 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:58.412134 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:58.413350 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:45:58.414014 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:45:58.414126 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:45:58.415159 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:45:58.415165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:45:58.415170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:45:58.415187 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:45:58.415229 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:45:58.415247 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:45:58.415353 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:45:58.415501 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 11:45:58.415550 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:45:58.418853 22247 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18ae7d0b9434fc49cd6396210c2911a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260924 11:45:58.419147 22247 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:45:58.419212 22247 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18ae7d0b9434fc49cd6396210c2911a, State: Initialized, Role: FOLLOWER
I20260924 11:45:58.419661 22247 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18ae7d0b9434fc49cd6396210c2911a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260924 11:45:58.419831 22247 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:45:58.419915 22247 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:45:58.420039 22247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:45:58.422091 22247 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18ae7d0b9434fc49cd6396210c2911a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260924 11:45:58.422489 22247 leader_election.cc:302] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18ae7d0b9434fc49cd6396210c2911a; no voters:
I20260924 11:45:58.423045 22247 leader_election.cc:288] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:45:58.423144 22257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:45:58.424104 22257 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [term 1 LEADER]: Becoming Leader. State: Replica: c18ae7d0b9434fc49cd6396210c2911a, State: Running, Role: LEADER
I20260924 11:45:58.424621 22257 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18ae7d0b9434fc49cd6396210c2911a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260924 11:45:58.425140 22247 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:45:58.427235 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c18ae7d0b9434fc49cd6396210c2911a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18ae7d0b9434fc49cd6396210c2911a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } } }
I20260924 11:45:58.427656 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:58.427642 22259 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18ae7d0b9434fc49cd6396210c2911a. Latest consensus state: current_term: 1 leader_uuid: "c18ae7d0b9434fc49cd6396210c2911a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18ae7d0b9434fc49cd6396210c2911a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } } }
I20260924 11:45:58.427994 22259 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a [sys.catalog]: This master's current role is: LEADER
I20260924 11:45:58.433373 22266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:45:58.439484 22266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:45:58.449265 22266 catalog_manager.cc:1347] Generated new cluster ID: 12cd1af4861b4e94a55a0cac40431012
I20260924 11:45:58.449362 22266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:45:58.458787 22266 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:45:58.460479 22266 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:45:58.469029 22266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18ae7d0b9434fc49cd6396210c2911a: Generated new TSK 0
I20260924 11:45:58.470036 22266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:45:58.484668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22180
W20260924 11:45:58.485772 22242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42559
W20260924 11:45:58.486568 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22180: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:45:58.486706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22180
W20260924 11:45:58.502766 22165 connection.cc:535] client connection to 127.0.97.62:40723 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40723): Connection reset by peer (error 104)
2026-09-24T11:45:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1745 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:45:58 dist-test-slave-qhrh krb5kdc[22278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:45:58 dist-test-slave-qhrh krb5kdc[22278](info): set up 2 sockets
Sep 24 11:45:58 dist-test-slave-qhrh krb5kdc[22278](info): commencing operation
krb5kdc: starting...
W20260924 11:45:59.516291 388 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:45:59 dist-test-slave-qhrh krb5kdc[22278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:45:59.733379 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45975
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:46:00.032624 22291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:00.032922 22291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:00.033113 22291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:00.040086 22291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:00.040252 22291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:00.040313 22291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:00.040369 22291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:00.040410 22291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:00.050873 22291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:00.052453 22291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:00.053782 22291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:00.064392 22296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:00.064875 22297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:00.065435 22299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:00.066390 22291 server_base.cc:1034] running on GCE node
I20260924 11:46:00.067093 22291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:00.068468 22291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:00.069689 22291 hybrid_clock.cc:648] HybridClock initialized: now 1790250360069649 us; error 50 us; skew 500 ppm
I20260924 11:46:00.070036 22291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:00 dist-test-slave-qhrh krb5kdc[22278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:00.081081 22291 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:46:00.082554 22291 webserver.cc:460] Webserver started at http://127.0.97.62:44809/ using document root <none> and password file <none>
I20260924 11:46:00.083256 22291 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:00.083401 22291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:00.083698 22291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:00.086616 22291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "2c9932a89fdb43129bda908a4c823722"
format_stamp: "Formatted at 2026-09-24 11:46:00 on dist-test-slave-qhrh"
server_key: "48340318c437d143fd217b775fb374b5"
server_key_iv: "6e9c87d748eb7a822221307fdb310f4d"
server_key_version: "encryptionkey@0"
I20260924 11:46:00.087368 22291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c9932a89fdb43129bda908a4c823722"
format_stamp: "Formatted at 2026-09-24 11:46:00 on dist-test-slave-qhrh"
server_key: "48340318c437d143fd217b775fb374b5"
server_key_iv: "6e9c87d748eb7a822221307fdb310f4d"
server_key_version: "encryptionkey@0"
I20260924 11:46:00.092221 22291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 11:46:00.095434 22306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:00.096722 22291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:46:00.096895 22291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "2c9932a89fdb43129bda908a4c823722"
format_stamp: "Formatted at 2026-09-24 11:46:00 on dist-test-slave-qhrh"
server_key: "48340318c437d143fd217b775fb374b5"
server_key_iv: "6e9c87d748eb7a822221307fdb310f4d"
server_key_version: "encryptionkey@0"
I20260924 11:46:00.097081 22291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:00.102298 22291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:00.106314 22291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:00.106609 22291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:00.123955 22291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45975
I20260924 11:46:00.124001 22357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45975 every 8 connection(s)
I20260924 11:46:00.125319 22291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:46:00.128720 22358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:00.131032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22291
I20260924 11:46:00.131202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:46:00.131610 388 external_mini_cluster.cc:1428] Setting key 621e2932ee1dfb69d70b515d75995e9f
I20260924 11:46:00.138932 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:00.133301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49263 (local address 127.0.97.62:45975)
0924 11:46:00.134117 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:00.134127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:00.134163 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:46:00.135715 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:00.135734 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:00.136095 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:00.136467 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:00.137486 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:00.137492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:00.137503 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:46:00.137516 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:00.137571 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:46:00.137591 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:00.137782 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:00.137893 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 11:46:00.138082 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":360,"threads_started":1}
I20260924 11:46:00.139719 22358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722: Bootstrap starting.
I20260924 11:46:00.142221 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:46:00.142825 22358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:00.143770 22358 log.cc:824] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:00.146761 22358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722: No bootstrap required, opened a new log
I20260924 11:46:00.149161 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:00.146229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60021 (local address 127.0.97.62:45975)
0924 11:46:00.146366 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:00.146370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:00.147334 (+ 964us) server_negotiation.cc:407] Connection header received
0924 11:46:00.147503 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:00.147513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:00.147555 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:00.147643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:00.148535 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:00.148540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:00.148545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:46:00.148564 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:00.148609 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:46:00.148625 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:00.148741 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:00.148848 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 11:46:00.148905 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 11:46:00.154053 22358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9932a89fdb43129bda908a4c823722" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260924 11:46:00.154356 22358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:46:00.154439 22358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9932a89fdb43129bda908a4c823722, State: Initialized, Role: FOLLOWER
I20260924 11:46:00.154955 22358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9932a89fdb43129bda908a4c823722" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260924 11:46:00.155170 22358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:46:00.155303 22358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:46:00.155465 22358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:00.157728 22358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9932a89fdb43129bda908a4c823722" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260924 11:46:00.158162 22358 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c9932a89fdb43129bda908a4c823722; no voters:
I20260924 11:46:00.158659 22358 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:46:00.158824 22368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:46:00.159567 22368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [term 1 LEADER]: Becoming Leader. State: Replica: 2c9932a89fdb43129bda908a4c823722, State: Running, Role: LEADER
I20260924 11:46:00.160218 22368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9932a89fdb43129bda908a4c823722" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } }
I20260924 11:46:00.160748 22358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:00.163177 22370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c9932a89fdb43129bda908a4c823722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9932a89fdb43129bda908a4c823722" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } } }
I20260924 11:46:00.163143 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9932a89fdb43129bda908a4c823722. Latest consensus state: current_term: 1 leader_uuid: "2c9932a89fdb43129bda908a4c823722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9932a89fdb43129bda908a4c823722" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45975 } } }
I20260924 11:46:00.163602 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:00.163599 22370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:00.164558 22377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:46:00.169539 22377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:46:00.177901 22377 catalog_manager.cc:1347] Generated new cluster ID: d570a92ced7e437ba3059b62e6a27e0e
I20260924 11:46:00.178011 22377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:46:00.200480 22377 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:46:00.200949 22377 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:46:00.209242 22377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c9932a89fdb43129bda908a4c823722: Generated new TSK 0
I20260924 11:46:00.210152 22377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:46:00.215054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22291
W20260924 11:46:00.216420 22353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49263
W20260924 11:46:00.217064 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22291: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:46:00.217195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22291
2026-09-24T11:46:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1727 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:46:00 dist-test-slave-qhrh krb5kdc[22389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:46:00 dist-test-slave-qhrh krb5kdc[22389](info): set up 2 sockets
Sep 24 11:46:00 dist-test-slave-qhrh krb5kdc[22389](info): commencing operation
krb5kdc: starting...
W20260924 11:46:01.260381 388 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:46:01 dist-test-slave-qhrh krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:46:01.490715 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34543
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:46:01.808036 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:01.808393 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:01.808564 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:01.815465 22402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:01.815586 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:01.815672 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:01.815788 22402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:01.815864 22402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:01.825242 22402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:01.826560 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:01.827872 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:01.839466 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:01.839438 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:01.839641 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:01.840617 22402 server_base.cc:1034] running on GCE node
I20260924 11:46:01.841435 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:01.842887 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:01.844199 22402 hybrid_clock.cc:648] HybridClock initialized: now 1790250361844083 us; error 94 us; skew 500 ppm
I20260924 11:46:01.844561 22402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:01 dist-test-slave-qhrh krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:01.847570 22402 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:46:01.849069 22402 webserver.cc:460] Webserver started at http://127.0.97.62:34551/ using document root <none> and password file <none>
I20260924 11:46:01.849766 22402 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:01.849916 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:01.850210 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:01.853202 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "81b8c9126c37409887a14d0700028192"
format_stamp: "Formatted at 2026-09-24 11:46:01 on dist-test-slave-qhrh"
server_key: "abe64c02caf4959fc7123cdc48b33e17"
server_key_iv: "140e395448c401ccc84727684742ed71"
server_key_version: "encryptionkey@0"
I20260924 11:46:01.853942 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "81b8c9126c37409887a14d0700028192"
format_stamp: "Formatted at 2026-09-24 11:46:01 on dist-test-slave-qhrh"
server_key: "abe64c02caf4959fc7123cdc48b33e17"
server_key_iv: "140e395448c401ccc84727684742ed71"
server_key_version: "encryptionkey@0"
I20260924 11:46:01.858956 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:46:01.862207 22417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:01.863412 22402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:46:01.863582 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "81b8c9126c37409887a14d0700028192"
format_stamp: "Formatted at 2026-09-24 11:46:01 on dist-test-slave-qhrh"
server_key: "abe64c02caf4959fc7123cdc48b33e17"
server_key_iv: "140e395448c401ccc84727684742ed71"
server_key_version: "encryptionkey@0"
I20260924 11:46:01.863755 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:01.884333 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:01.888059 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:01.888339 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:01.906904 22402 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34543
I20260924 11:46:01.906927 22468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34543 every 8 connection(s)
I20260924 11:46:01.908238 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:46:01.911734 22469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:01.912971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22402
I20260924 11:46:01.913170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:46:01.913625 388 external_mini_cluster.cc:1428] Setting key 81cc6628e0debfb5ed3816f66299143d
I20260924 11:46:01.925457 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192: Bootstrap starting.
Sep 24 11:46:01 dist-test-slave-qhrh krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:46:01.928515 22469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:01.929939 22469 log.cc:824] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:01.933293 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192: No bootstrap required, opened a new log
I20260924 11:46:01.936925 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:01.915443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51805 (local address 127.0.97.62:34543)
0924 11:46:01.916266 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:01.916281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:01.916330 (+ 49us) server_negotiation.cc:407] Connection header received
0924 11:46:01.918158 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:01.918184 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:01.918667 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:01.919094 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:01.920314 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:01.922257 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:01.923527 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:01.924536 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:01.924705 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:01.928706 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:01.928716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:01.928736 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:01.928784 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:01.932689 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:01.933335 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:01.933345 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:01.933352 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:01.933546 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:01.933930 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:01.933937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:01.933942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:01.934546 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:01.934828 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:01.935305 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 11:46:01.935785 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":341,"threads_started":1}
I20260924 11:46:01.940502 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:46:01.941793 22469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8c9126c37409887a14d0700028192" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34543 } }
I20260924 11:46:01.942162 22469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:46:01.942266 22469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b8c9126c37409887a14d0700028192, State: Initialized, Role: FOLLOWER
I20260924 11:46:01.942817 22469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8c9126c37409887a14d0700028192" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34543 } }
I20260924 11:46:01.943118 22469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:46:01.943267 22469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:46:01.943447 22469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:01.946048 22469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8c9126c37409887a14d0700028192" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34543 } }
I20260924 11:46:01.946568 22469 leader_election.cc:302] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b8c9126c37409887a14d0700028192; no voters:
I20260924 11:46:01.947227 22469 leader_election.cc:288] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:46:01.947321 22479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:46:01.949679 22469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:01.954463 22479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [term 1 LEADER]: Becoming Leader. State: Replica: 81b8c9126c37409887a14d0700028192, State: Running, Role: LEADER
I20260924 11:46:01.955256 22479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8c9126c37409887a14d0700028192" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34543 } }
I20260924 11:46:01.958508 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b8c9126c37409887a14d0700028192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8c9126c37409887a14d0700028192" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34543 } } }
I20260924 11:46:01.958995 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:01.959839 22481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b8c9126c37409887a14d0700028192. Latest consensus state: current_term: 1 leader_uuid: "81b8c9126c37409887a14d0700028192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8c9126c37409887a14d0700028192" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34543 } } }
I20260924 11:46:01.960278 22481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:01.960446 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:01.944914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42155 (local address 127.0.97.62:34543)
0924 11:46:01.945094 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:01.945105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:01.945470 (+ 365us) server_negotiation.cc:407] Connection header received
0924 11:46:01.945648 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:01.945653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:01.945727 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:01.945828 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:01.947273 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:01.948463 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:01.949759 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:01.950242 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:01.950379 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:01.953143 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:01.953150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:01.953160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:01.953189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:01.956311 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:01.957010 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:01.957017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:01.957021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:01.957197 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:01.957703 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:01.957707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:01.957710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:01.958157 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:01.959375 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:01.959556 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 11:46:01.959820 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:46:01.964895 22489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:46:01.970453 22489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:46:01.979161 22489 catalog_manager.cc:1347] Generated new cluster ID: a1e9ee13b07c4d17a4e7420e7f0d5ca6
I20260924 11:46:01.979259 22489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:46:02.006575 22489 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:46:02.007987 22489 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:46:02.013540 22489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b8c9126c37409887a14d0700028192: Generated new TSK 0
I20260924 11:46:02.014451 22489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:46:02.030968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22402
I20260924 11:46:02.031960 22464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:02.123452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22402
2026-09-24T11:46:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:46:02 dist-test-slave-qhrh krb5kdc[22502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:46:02 dist-test-slave-qhrh krb5kdc[22502](info): set up 2 sockets
Sep 24 11:46:02 dist-test-slave-qhrh krb5kdc[22502](info): commencing operation
krb5kdc: starting...
W20260924 11:46:03.176437 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:46:03 dist-test-slave-qhrh krb5kdc[22502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:46:03.411407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34825
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:46:03.716838 22515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:03.717092 22515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:03.717280 22515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:03.724033 22515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:03.724126 22515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:03.724248 22515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:03.724364 22515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:03.724458 22515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:03.733796 22515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:03.734956 22515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:03.736209 22515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:03.746752 22521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:03.746848 22520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:03.746898 22523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:03.747666 22515 server_base.cc:1034] running on GCE node
I20260924 11:46:03.748360 22515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:03.749557 22515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:03.750734 22515 hybrid_clock.cc:648] HybridClock initialized: now 1790250363750693 us; error 30 us; skew 500 ppm
I20260924 11:46:03.751050 22515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:03 dist-test-slave-qhrh krb5kdc[22502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:03.761328 22515 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:46:03.762727 22515 webserver.cc:460] Webserver started at http://127.0.97.62:38437/ using document root <none> and password file <none>
I20260924 11:46:03.763372 22515 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:03.763522 22515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:03.763828 22515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:03.766702 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "056c881322604f1fa0787d09a84405a8"
format_stamp: "Formatted at 2026-09-24 11:46:03 on dist-test-slave-qhrh"
server_key: "a4f43134d458f39d1abfd13f0122f01c"
server_key_iv: "4dd47ceaa94569635e482ec4347e3e1c"
server_key_version: "encryptionkey@0"
I20260924 11:46:03.767439 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "056c881322604f1fa0787d09a84405a8"
format_stamp: "Formatted at 2026-09-24 11:46:03 on dist-test-slave-qhrh"
server_key: "a4f43134d458f39d1abfd13f0122f01c"
server_key_iv: "4dd47ceaa94569635e482ec4347e3e1c"
server_key_version: "encryptionkey@0"
I20260924 11:46:03.772113 22515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 11:46:03.775177 22530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:03.776413 22515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:46:03.776602 22515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "056c881322604f1fa0787d09a84405a8"
format_stamp: "Formatted at 2026-09-24 11:46:03 on dist-test-slave-qhrh"
server_key: "a4f43134d458f39d1abfd13f0122f01c"
server_key_iv: "4dd47ceaa94569635e482ec4347e3e1c"
server_key_version: "encryptionkey@0"
I20260924 11:46:03.776783 22515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:03.789423 22515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:03.795400 22515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:03.795670 22515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:03.812834 22515 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34825
I20260924 11:46:03.812867 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34825 every 8 connection(s)
I20260924 11:46:03.814096 22515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:46:03.817528 22582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:03.821229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22515
I20260924 11:46:03.821390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:46:03.821830 388 external_mini_cluster.cc:1428] Setting key 8ede1b1efe72d9b73095fb152b08da36
I20260924 11:46:03.827656 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8: Bootstrap starting.
I20260924 11:46:03.830971 22582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:03.832130 22582 log.cc:824] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:03.835059 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8: No bootstrap required, opened a new log
I20260924 11:46:03.835637 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:03.823445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51765 (local address 127.0.97.62:34825)
0924 11:46:03.824221 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:03.824232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:03.824269 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:46:03.826284 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:03.826304 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:03.826647 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:03.826947 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:03.827518 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:03.828904 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:03.830729 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:03.831410 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:03.831538 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:03.833587 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:03.833593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:03.833604 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:46:03.834108 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:03.834171 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:46:03.834193 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:03.834280 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:03.834516 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 11:46:03.834743 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":317,"threads_started":1}
I20260924 11:46:03.839039 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:46:03.842631 22582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056c881322604f1fa0787d09a84405a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34825 } }
I20260924 11:46:03.843057 22582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:46:03.843173 22582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056c881322604f1fa0787d09a84405a8, State: Initialized, Role: FOLLOWER
I20260924 11:46:03.843740 22582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056c881322604f1fa0787d09a84405a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34825 } }
I20260924 11:46:03.843966 22582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:46:03.844084 22582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:46:03.844261 22582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:03.846239 22582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056c881322604f1fa0787d09a84405a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34825 } }
I20260924 11:46:03.846717 22582 leader_election.cc:302] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 056c881322604f1fa0787d09a84405a8; no voters:
I20260924 11:46:03.847285 22582 leader_election.cc:288] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:46:03.847337 22592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:46:03.848173 22592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [term 1 LEADER]: Becoming Leader. State: Replica: 056c881322604f1fa0787d09a84405a8, State: Running, Role: LEADER
I20260924 11:46:03.849044 22592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056c881322604f1fa0787d09a84405a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34825 } }
I20260924 11:46:03.849807 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:03.842832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59015 (local address 127.0.97.62:34825)
0924 11:46:03.843016 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:03.843023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:03.843563 (+ 540us) server_negotiation.cc:407] Connection header received
0924 11:46:03.843720 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:03.843725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:03.843792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:03.843888 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:03.844427 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:03.845111 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:03.846282 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:03.846783 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:03.846937 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:03.848854 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:03.848859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:03.848863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:46:03.848879 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:03.848932 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:46:03.848954 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:03.849027 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:03.849442 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 11:46:03.849521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:46:03.849992 22582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:03.851521 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "056c881322604f1fa0787d09a84405a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056c881322604f1fa0787d09a84405a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34825 } } }
I20260924 11:46:03.851864 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:03.852407 22594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 056c881322604f1fa0787d09a84405a8. Latest consensus state: current_term: 1 leader_uuid: "056c881322604f1fa0787d09a84405a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056c881322604f1fa0787d09a84405a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34825 } } }
I20260924 11:46:03.852694 22594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:03.852837 22600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:46:03.859747 22600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:46:03.873075 22600 catalog_manager.cc:1347] Generated new cluster ID: ba5a2d4f0d044e3db14df1ecd95aaa4b
I20260924 11:46:03.873199 22600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:46:03.884815 22600 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:46:03.885902 22600 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:46:03.892269 22600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 056c881322604f1fa0787d09a84405a8: Generated new TSK 0
I20260924 11:46:03.893033 22600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:46:03.907585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22515
W20260924 11:46:03.908658 22577 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51765
W20260924 11:46:03.909294 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:46:03.909428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22515
2026-09-24T11:46:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1784 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:46:04 dist-test-slave-qhrh krb5kdc[22613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:46:04 dist-test-slave-qhrh krb5kdc[22613](info): set up 2 sockets
Sep 24 11:46:04 dist-test-slave-qhrh krb5kdc[22613](info): commencing operation
krb5kdc: starting...
W20260924 11:46:04.956516 388 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:46:05 dist-test-slave-qhrh krb5kdc[22613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:46:05.179560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44927
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:46:05.490243 22626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:05.490509 22626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:05.490645 22626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:05.497139 22626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:05.497232 22626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:05.497336 22626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:05.497443 22626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:05.497521 22626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:05.506558 22626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:05.507704 22626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:05.508939 22626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:05.520344 22634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:05.520413 22631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:05.520473 22632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:05.521093 22626 server_base.cc:1034] running on GCE node
I20260924 11:46:05.521799 22626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:05.523078 22626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:05.524261 22626 hybrid_clock.cc:648] HybridClock initialized: now 1790250365524218 us; error 30 us; skew 500 ppm
I20260924 11:46:05.524561 22626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:05 dist-test-slave-qhrh krb5kdc[22613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:05.539716 22626 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:46:05.541154 22626 webserver.cc:460] Webserver started at http://127.0.97.62:39625/ using document root <none> and password file <none>
I20260924 11:46:05.541797 22626 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:05.541903 22626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:05.542210 22626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:05.545039 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "3aa560c1ab8a4535a8594f6fac9998d3"
format_stamp: "Formatted at 2026-09-24 11:46:05 on dist-test-slave-qhrh"
server_key: "da7d58afc7d2ea12a1cd31db8fdadf51"
server_key_iv: "30f7722016bab614949b3c5354eee234"
server_key_version: "encryptionkey@0"
I20260924 11:46:05.545773 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "3aa560c1ab8a4535a8594f6fac9998d3"
format_stamp: "Formatted at 2026-09-24 11:46:05 on dist-test-slave-qhrh"
server_key: "da7d58afc7d2ea12a1cd31db8fdadf51"
server_key_iv: "30f7722016bab614949b3c5354eee234"
server_key_version: "encryptionkey@0"
I20260924 11:46:05.550527 22626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:46:05.553659 22641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:05.554791 22626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:46:05.554963 22626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "3aa560c1ab8a4535a8594f6fac9998d3"
format_stamp: "Formatted at 2026-09-24 11:46:05 on dist-test-slave-qhrh"
server_key: "da7d58afc7d2ea12a1cd31db8fdadf51"
server_key_iv: "30f7722016bab614949b3c5354eee234"
server_key_version: "encryptionkey@0"
I20260924 11:46:05.555140 22626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:05.560395 22626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:05.563961 22626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:05.564239 22626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:05.581044 22626 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44927
I20260924 11:46:05.581081 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44927 every 8 connection(s)
I20260924 11:46:05.582327 22626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:46:05.585615 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:05.590564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22626
I20260924 11:46:05.590729 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:46:05.591144 388 external_mini_cluster.cc:1428] Setting key f0577285edf8c0388be71bf1a5f0f57b
I20260924 11:46:05.596678 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3: Bootstrap starting.
I20260924 11:46:05.599493 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:05.592845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59757 (local address 127.0.97.62:44927)
0924 11:46:05.593645 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:05.593659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:05.593725 (+ 66us) server_negotiation.cc:407] Connection header received
0924 11:46:05.595938 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:05.595971 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:05.596367 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:05.596787 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:05.597695 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:05.597702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:05.597719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:46:05.597748 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:05.597813 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:46:05.597847 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:05.598048 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:05.598168 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 11:46:05.598371 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":310,"threads_started":1}
I20260924 11:46:05.600276 22693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:05.601609 22693 log.cc:824] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:05.603230 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:46:05.604378 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3: No bootstrap required, opened a new log
I20260924 11:46:05.609526 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:05.607156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34513 (local address 127.0.97.62:44927)
0924 11:46:05.607318 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:05.607325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:05.607733 (+ 408us) server_negotiation.cc:407] Connection header received
0924 11:46:05.607935 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:05.607939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:05.607987 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:05.608090 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:05.608957 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:05.608962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:05.608966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:46:05.608981 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:05.609028 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:46:05.609044 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:05.609152 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:05.609242 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 11:46:05.609295 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:46:05.612572 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44927 } }
I20260924 11:46:05.612897 22693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:46:05.612995 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa560c1ab8a4535a8594f6fac9998d3, State: Initialized, Role: FOLLOWER
I20260924 11:46:05.613648 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44927 } }
I20260924 11:46:05.613915 22693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:46:05.614040 22693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:46:05.614156 22693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:05.615978 22693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44927 } }
I20260924 11:46:05.616434 22693 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa560c1ab8a4535a8594f6fac9998d3; no voters:
I20260924 11:46:05.616945 22693 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:46:05.617060 22703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:46:05.617748 22703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa560c1ab8a4535a8594f6fac9998d3, State: Running, Role: LEADER
I20260924 11:46:05.618227 22703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44927 } }
I20260924 11:46:05.619087 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:05.620911 22705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa560c1ab8a4535a8594f6fac9998d3. Latest consensus state: current_term: 1 leader_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44927 } } }
I20260924 11:46:05.620911 22704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa560c1ab8a4535a8594f6fac9998d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44927 } } }
I20260924 11:46:05.621345 22705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:05.621345 22704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:05.623353 22709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:46:05.627419 22709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:46:05.636824 22709 catalog_manager.cc:1347] Generated new cluster ID: 5591480401184501b1a6d9cdebe9dcd5
I20260924 11:46:05.636931 22709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:46:05.648034 22709 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:46:05.648461 22709 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:46:05.655396 22709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aa560c1ab8a4535a8594f6fac9998d3: Generated new TSK 0
I20260924 11:46:05.656041 22709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:46:05.661105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22626
W20260924 11:46:05.662345 22688 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59757
W20260924 11:46:05.662990 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22626: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:46:05.663123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22626
2026-09-24T11:46:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1753 ms)
[----------] 18 tests from AuthTokenIssuingTest (32385 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:46:05 dist-test-slave-qhrh krb5kdc[22724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:46:05 dist-test-slave-qhrh krb5kdc[22724](info): set up 2 sockets
Sep 24 11:46:05 dist-test-slave-qhrh krb5kdc[22724](info): commencing operation
krb5kdc: starting...
W20260924 11:46:06.696417 388 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:46:06 dist-test-slave-qhrh krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:46:06.921624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41611
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41611,127.0.97.61:34297,127.0.97.60:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:46:07.227708 22737 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:46:07.228024 22737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:07.228178 22737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:07.228322 22737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:07.235118 22737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:07.235188 22737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:07.235237 22737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:07.235275 22737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:07.235355 22737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:07.243963 22737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41611,127.0.97.61:34297,127.0.97.60:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:07.245102 22737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:07.246232 22737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:07.257150 22743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:07.257239 22742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:07.257248 22745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:07.257790 22737 server_base.cc:1034] running on GCE node
I20260924 11:46:07.258462 22737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:07.259900 22737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:07.261173 22737 hybrid_clock.cc:648] HybridClock initialized: now 1790250367261117 us; error 39 us; skew 500 ppm
I20260924 11:46:07.261497 22737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:07 dist-test-slave-qhrh krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:07.264405 22737 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:46:07.265682 22737 webserver.cc:460] Webserver started at http://127.0.97.62:40303/ using document root <none> and password file <none>
I20260924 11:46:07.266314 22737 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:07.266431 22737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:07.266716 22737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:07.269516 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "47adb81c82a94f6abe683a0a0529df4c"
format_stamp: "Formatted at 2026-09-24 11:46:07 on dist-test-slave-qhrh"
server_key: "e3638b47943e6d521b7596fb8008d0ba"
server_key_iv: "9c2c97d6156e7034c712c4f1db660c37"
server_key_version: "encryptionkey@0"
I20260924 11:46:07.270257 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "47adb81c82a94f6abe683a0a0529df4c"
format_stamp: "Formatted at 2026-09-24 11:46:07 on dist-test-slave-qhrh"
server_key: "e3638b47943e6d521b7596fb8008d0ba"
server_key_iv: "9c2c97d6156e7034c712c4f1db660c37"
server_key_version: "encryptionkey@0"
I20260924 11:46:07.274762 22737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:46:07.277935 22752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:07.279065 22737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:46:07.279217 22737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "47adb81c82a94f6abe683a0a0529df4c"
format_stamp: "Formatted at 2026-09-24 11:46:07 on dist-test-slave-qhrh"
server_key: "e3638b47943e6d521b7596fb8008d0ba"
server_key_iv: "9c2c97d6156e7034c712c4f1db660c37"
server_key_version: "encryptionkey@0"
I20260924 11:46:07.279395 22737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:07.300431 22737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:07.303792 22737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:07.304028 22737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:07.321161 22737 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41611
I20260924 11:46:07.321182 22803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41611 every 8 connection(s)
I20260924 11:46:07.322342 22737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:46:07.325608 22804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:07.331497 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:07.332438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22737
I20260924 11:46:07.332638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:46:07.333386 388 external_mini_cluster.cc:1428] Setting key c949a16dbe144778315fbcd1aa22fa90
Sep 24 11:46:07 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:46:07.351574 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.334687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.0.97.62:41611)
0924 11:46:07.336187 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:07.336195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:07.336242 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:46:07.337666 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:07.337671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:07.337871 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:07.338078 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:07.339199 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.340335 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.341510 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.342003 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.342187 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:07.345548 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:07.345555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:07.345563 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:07.345586 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:07.348047 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:07.348829 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:07.348834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:07.348836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:07.348951 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:07.349335 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:07.349337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:07.349339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:07.349711 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:07.349846 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:07.350414 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 11:46:07.350676 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1295,"thread_start_us":1119,"threads_started":1}
I20260924 11:46:07.351609 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.334687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41611 (local address 127.0.0.1:38314)
0924 11:46:07.335430 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.335506 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:07.337313 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:07.338088 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:07.338094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:07.338717 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:07.339006 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:07.339021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.340460 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.340464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:07.341381 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:07.341392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.342220 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.342244 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:07.342429 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:07.343055 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:07.343074 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:07.345359 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:07.348182 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:07.348194 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:07.348206 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:07.348717 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:07.349046 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:07.349053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:07.349058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:07.349239 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:07.349837 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:07.349845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:07.350057 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 11:46:07.350311 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 11:46:07.350845 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":64,"thread_start_us":329,"threads_started":1}
I20260924 11:46:07.355085 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:07.356021 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.355677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:39440)
0924 11:46:07.355813 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.355928 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:46:07.356317 22755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260924 11:46:07.357697 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.357398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:39442)
0924 11:46:07.357528 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.357597 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:46:07.358013 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34297: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 11:46:07.383445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34297
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41611,127.0.97.61:34297,127.0.97.60:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:46:07.407294 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } attempt: 1
I20260924 11:46:07.408533 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.408054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:39458)
0924 11:46:07.408287 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.408366 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 11:46:07.409571 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.409231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:39466)
0924 11:46:07.409376 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.409439 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:46:07.409826 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34297: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
I20260924 11:46:07.478102 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } attempt: 2
I20260924 11:46:07.479274 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.478802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:39474)
0924 11:46:07.479070 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.479150 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:46:07.480226 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.479882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:39478)
0924 11:46:07.480012 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.480075 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:46:07.480437 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34297: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
I20260924 11:46:07.571707 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } attempt: 3
I20260924 11:46:07.572932 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.572520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45576)
0924 11:46:07.572713 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.572800 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 11:46:07.573941 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.573649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45580)
0924 11:46:07.573768 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.573827 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:46:07.574206 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34297: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
W20260924 11:46:07.689646 22811 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:46:07.689934 22811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:07.689993 22811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:07.690114 22811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:07.696401 22811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:07.696470 22811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:07.696589 22811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:07.696681 22811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:07.696772 22811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:07.705570 22811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41611,127.0.97.61:34297,127.0.97.60:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:07.706692 22811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:07.707911 22811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:46:07.717455 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } attempt: 4
W20260924 11:46:07.718717 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:07.718690 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:07.718874 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.718238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45594)
0924 11:46:07.718605 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.718705 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260924 11:46:07.718777 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:07.720000 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.719551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45598)
0924 11:46:07.719697 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.719777 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:46:07.720232 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34297: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34297: connect: Connection refused (error 111)
I20260924 11:46:07.720428 22811 server_base.cc:1034] running on GCE node
I20260924 11:46:07.721197 22811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:07.722442 22811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:07.723680 22811 hybrid_clock.cc:648] HybridClock initialized: now 1790250367723620 us; error 36 us; skew 500 ppm
I20260924 11:46:07.723979 22811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:07 dist-test-slave-qhrh krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:07.726972 22811 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 11:46:07.728310 22811 webserver.cc:460] Webserver started at http://127.0.97.61:38945/ using document root <none> and password file <none>
I20260924 11:46:07.728948 22811 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:07.729071 22811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:07.729373 22811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:07.732019 22811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data/instance:
uuid: "328cdb7e8c794a72af5a6d90c25dfd17"
format_stamp: "Formatted at 2026-09-24 11:46:07 on dist-test-slave-qhrh"
server_key: "1e37f5d295ef77a39ed074887744e74b"
server_key_iv: "9880d924d585067445efb779251cc60b"
server_key_version: "encryptionkey@0"
I20260924 11:46:07.732797 22811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal/instance:
uuid: "328cdb7e8c794a72af5a6d90c25dfd17"
format_stamp: "Formatted at 2026-09-24 11:46:07 on dist-test-slave-qhrh"
server_key: "1e37f5d295ef77a39ed074887744e74b"
server_key_iv: "9880d924d585067445efb779251cc60b"
server_key_version: "encryptionkey@0"
I20260924 11:46:07.737469 22811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 11:46:07.740453 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:07.741675 22811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:46:07.741842 22811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal
uuid: "328cdb7e8c794a72af5a6d90c25dfd17"
format_stamp: "Formatted at 2026-09-24 11:46:07 on dist-test-slave-qhrh"
server_key: "1e37f5d295ef77a39ed074887744e74b"
server_key_iv: "9880d924d585067445efb779251cc60b"
server_key_version: "encryptionkey@0"
I20260924 11:46:07.742036 22811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:07.759992 22811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:07.763530 22811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:07.763784 22811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:07.780850 22811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34297
I20260924 11:46:07.780869 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34297 every 8 connection(s)
I20260924 11:46:07.782167 22811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
I20260924 11:46:07.784265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22811
I20260924 11:46:07.784437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-1/wal/instance
I20260924 11:46:07.784822 388 external_mini_cluster.cc:1428] Setting key 341ddff8bfc55d89b4fa5ea25d6ecd61
I20260924 11:46:07.785775 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:07.792013 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:07 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 11:46:07.812718 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.794991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.0.97.62:41611)
0924 11:46:07.795219 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:07.795227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:07.796266 (+ 1039us) server_negotiation.cc:407] Connection header received
0924 11:46:07.798112 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:07.798119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:07.798203 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:07.798309 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:07.799594 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.800708 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.802136 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.802992 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.803132 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:07.806574 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:07.806582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:07.806587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:07.806627 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:07.809456 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:07.810332 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:07.810336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:07.810338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:07.810492 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:07.810920 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:07.810925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:07.810927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:07.811308 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:07.811454 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:07.812063 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 11:46:07.812308 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:46:07.813277 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.795404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41611 (local address 127.0.0.1:60786)
0924 11:46:07.796095 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.796162 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:07.797815 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:07.798475 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:07.798491 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:07.798960 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:07.799424 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:07.799439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.800846 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.800850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:07.801975 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:07.801987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.803180 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.803184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:07.803364 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:07.804041 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:07.804061 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:07.806328 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:07.809697 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:07.809705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:07.809719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:07.810170 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:07.810608 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:07.810611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:07.810614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:07.810779 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:07.811419 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:07.811424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:07.811718 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 11:46:07.811932 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 11:46:07.812309 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":300,"threads_started":1}
I20260924 11:46:07.814906 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:07 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:46:07.829667 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.816016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.0.97.61:34297)
0924 11:46:07.816590 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:07.816595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:07.816616 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:46:07.816670 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:07.816675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:07.816832 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:07.817002 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:07.817690 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.819076 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.820307 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.820837 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.820987 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:07.824099 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:07.824109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:07.824120 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:07.824194 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:07.826855 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:07.827481 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:07.827489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:07.827495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:07.827605 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:07.827950 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:07.827958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:07.827961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:07.828391 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:07.828574 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:07.829157 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 11:46:07.829366 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":314,"threads_started":1}
I20260924 11:46:07.829788 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.815706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45606)
0924 11:46:07.815931 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.815946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:07.816021 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:07.816983 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:07.816987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:07.817403 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:07.817530 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:07.817537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.819212 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.819217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:07.819999 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:07.820008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:07.820991 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:07.820996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:07.821170 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:07.821756 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:07.821771 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:07.823893 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:07.826999 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:07.827006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:07.827009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:07.827356 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:07.827714 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:07.827718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:07.827722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:07.827848 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:07.828606 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:07.828614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:07.828791 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 11:46:07.828968 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 11:46:07.829279 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 11:46:07.831988 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:07.833022 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.832622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35354)
0924 11:46:07.832792 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.832891 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:46:07.833271 22828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
I20260924 11:46:07.834721 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.834403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35364)
0924 11:46:07.834547 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.834632 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:46:07.834906 22878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 11:46:07.839454 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37253
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41611,127.0.97.61:34297,127.0.97.60:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:46:07.884220 22878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 1
I20260924 11:46:07.885246 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.884842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35376)
0924 11:46:07.885017 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.885112 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 11:46:07.886118 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.885834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35386)
0924 11:46:07.885958 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.886032 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 11:46:07.886281 22878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
I20260924 11:46:07.954558 22878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 2
I20260924 11:46:07.955812 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.955164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35398)
0924 11:46:07.955424 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.955614 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 11:46:07.956877 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:07.956482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35406)
0924 11:46:07.956637 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:07.956717 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:46:07.957120 22878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
I20260924 11:46:08.019464 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } attempt: 5
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:46:08.033830 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.020099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45614)
0924 11:46:08.020489 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.020510 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:08.020621 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:08.021115 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:08.021118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:08.021460 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:08.021642 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.021650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.023224 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.023227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.024033 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.024042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.025053 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.025057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.025199 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.025827 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:08.025844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:08.028071 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:08.031675 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.031678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.031681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.032022 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.032412 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.032415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.032417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.032537 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.033179 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:08.033182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:08.033303 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 11:46:08.033415 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 11:46:08.033598 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260924 11:46:08.034144 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.020439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.0.97.61:34297)
0924 11:46:08.020713 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.020722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.020748 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:46:08.020827 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.020835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.020952 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.021097 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.021827 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.023086 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.024210 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.024913 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.025043 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.028292 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.028298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.028302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.028340 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.031556 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.032135 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.032161 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.032165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.032305 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.032648 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.032656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.032661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.033068 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.033214 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.033561 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.033768 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 11:46:08.035107 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:08.036190 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.035779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35412)
0924 11:46:08.035933 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.036017 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:46:08.037212 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.036889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35424)
0924 11:46:08.037042 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.037119 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:46:08.037416 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
I20260924 11:46:08.048332 22878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 3
I20260924 11:46:08.049407 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.048818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35440)
0924 11:46:08.049056 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.049191 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 11:46:08.050468 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.050038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35456)
0924 11:46:08.050249 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.050340 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260924 11:46:08.050747 22878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
I20260924 11:46:08.088647 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 1
I20260924 11:46:08.089843 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.089390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35464)
0924 11:46:08.089583 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.089675 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:46:08.090863 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.090540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35478)
0924 11:46:08.090693 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.090760 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:46:08.091101 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
W20260924 11:46:08.158643 22885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:46:08.158977 22885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:08.159097 22885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:08.159224 22885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:46:08.159322 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 2
I20260924 11:46:08.160499 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.160014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35484)
0924 11:46:08.160254 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.160338 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:46:08.161482 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.161159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35494)
0924 11:46:08.161307 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.161371 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 11:46:08.161701 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
W20260924 11:46:08.166646 22885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:08.166757 22885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:08.166899 22885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:08.166989 22885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:08.167097 22885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:08.176414 22885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41611,127.0.97.61:34297,127.0.97.60:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:08.177645 22885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:08.178930 22885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:08.189795 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:08.189934 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:08.189977 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:08.190757 22885 server_base.cc:1034] running on GCE node
I20260924 11:46:08.191478 22885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:08.192848 22885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:08.193949 22878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 4
I20260924 11:46:08.194099 22885 hybrid_clock.cc:648] HybridClock initialized: now 1790250368194023 us; error 35 us; skew 500 ppm
I20260924 11:46:08.194500 22885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:46:08.194983 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.194542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35498)
0924 11:46:08.194771 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.194841 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:46:08.196007 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.195700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35510)
0924 11:46:08.195838 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.195901 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:46:08.196271 22878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37253: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37253: connect: Connection refused (error 111)
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:08.197891 22885 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 11:46:08.199313 22885 webserver.cc:460] Webserver started at http://127.0.97.60:40471/ using document root <none> and password file <none>
I20260924 11:46:08.199904 22885 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:08.199985 22885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:08.200335 22885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:08.203198 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data/instance:
uuid: "cfd016c43f964122a123ea4421101202"
format_stamp: "Formatted at 2026-09-24 11:46:08 on dist-test-slave-qhrh"
server_key: "028bd4a5f13310039ed8a813af6b2192"
server_key_iv: "235d8f38ec01054b1743c710d8af3d4a"
server_key_version: "encryptionkey@0"
I20260924 11:46:08.203878 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal/instance:
uuid: "cfd016c43f964122a123ea4421101202"
format_stamp: "Formatted at 2026-09-24 11:46:08 on dist-test-slave-qhrh"
server_key: "028bd4a5f13310039ed8a813af6b2192"
server_key_iv: "235d8f38ec01054b1743c710d8af3d4a"
server_key_version: "encryptionkey@0"
I20260924 11:46:08.208674 22885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:46:08.211997 22900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:08.213281 22885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:46:08.213397 22885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal
uuid: "cfd016c43f964122a123ea4421101202"
format_stamp: "Formatted at 2026-09-24 11:46:08 on dist-test-slave-qhrh"
server_key: "028bd4a5f13310039ed8a813af6b2192"
server_key_iv: "235d8f38ec01054b1743c710d8af3d4a"
server_key_version: "encryptionkey@0"
I20260924 11:46:08.213559 22885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:08.245267 22885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:08.249646 22885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:08.249857 22885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:08.267612 22885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37253
I20260924 11:46:08.267643 22951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37253 every 8 connection(s)
I20260924 11:46:08.267884 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 3
I20260924 11:46:08.269219 22885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
I20260924 11:46:08.271786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 22885
I20260924 11:46:08.271965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250027667721-388-0/minicluster-data/master-2/wal/instance
I20260924 11:46:08.272445 388 external_mini_cluster.cc:1428] Setting key 28a1fe8fdb193a29b4f2823985410bb8
I20260924 11:46:08.273941 22952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:46:08.283524 22952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:08.292236 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.268512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35524)
0924 11:46:08.268723 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.268741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:08.268858 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:08.273756 (+ 4898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:08.273762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:08.274273 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:08.274481 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.274494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.277363 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.277369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.278931 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.278946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.280292 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.280298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.280481 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.281167 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:08.281189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:08.284128 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:08.288800 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.288807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.288811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.289332 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.289875 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.289879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.289882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.290063 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.291136 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:08.291141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:08.291297 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:46:08.291479 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 11:46:08.291758 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:46:08.293529 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.269510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.0.97.60:37253)
0924 11:46:08.270465 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.270479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.270557 (+ 78us) server_negotiation.cc:407] Connection header received
0924 11:46:08.272825 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.272855 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.273308 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.274709 (+ 1401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.275183 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.277175 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.279134 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.280094 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.280830 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.284427 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.284436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.284452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.284490 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.288600 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.289529 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.289537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.289545 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.289737 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.290223 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.290228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.290231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.290978 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.291158 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.291747 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.292177 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":673,"thread_start_us":426,"threads_started":1}
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:46:08.301407 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c: Bootstrap starting.
I20260924 11:46:08.303753 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.287048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.0.97.62:41611)
0924 11:46:08.287237 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.287243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.287480 (+ 237us) server_negotiation.cc:407] Connection header received
0924 11:46:08.287684 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.287692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.287770 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.287872 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.289344 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.290041 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.291745 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.292484 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.292583 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.296885 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.296891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.296914 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.296949 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.300526 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.301383 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.301393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.301397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.301587 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.302024 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.302029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.302032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.302413 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.302571 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.303164 (+ 593us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.303373 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:46:08.303817 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.286543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41611 (local address 127.0.0.1:60792)
0924 11:46:08.287354 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.287392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:08.287561 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:08.288021 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:08.288029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:08.288950 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:08.289197 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.289205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.290199 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.290204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.291597 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.291609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.292731 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.292738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.292891 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.293568 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:08.293589 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:08.296592 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:08.300672 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.300683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.300690 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.301256 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.301702 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.301712 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.301716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.301895 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.302565 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:08.302575 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:08.302777 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 11:46:08.303021 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 11:46:08.303367 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":2,"thread_start_us":413,"threads_started":1}
I20260924 11:46:08.305387 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.273397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.97.60:37253)
0924 11:46:08.287895 (+ 14498us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.287901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.288025 (+ 124us) server_negotiation.cc:407] Connection header received
0924 11:46:08.288251 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.288256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.288317 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.288430 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.290232 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.291545 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.293564 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.294244 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.294375 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.298801 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.298806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.298809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.298880 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.302133 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.302977 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.302985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.302991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.303151 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.303634 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.303638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.303641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.303990 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.304368 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.304803 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.304996 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":14365,"thread_start_us":12925,"threads_started":1}
I20260924 11:46:08.305426 22952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:08.306289 22804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:08.307518 22804 log.cc:824] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:08.310711 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c: No bootstrap required, opened a new log
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:46:08.320731 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.306218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.0.97.61:34297)
0924 11:46:08.306518 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.306523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.306796 (+ 273us) server_negotiation.cc:407] Connection header received
0924 11:46:08.307065 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.307074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.307154 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.307211 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.308023 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.309096 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.310422 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.311058 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.311171 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.314689 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.314695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.314698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.314728 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.317789 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.318530 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.318536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.318539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.318719 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.319111 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.319118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.319123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.319496 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.319667 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.320051 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.320287 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260924 11:46:08.320925 22804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.321321 22804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47adb81c82a94f6abe683a0a0529df4c, State: Initialized, Role: FOLLOWER
I20260924 11:46:08.321800 22804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.322865 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.323354 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:08.324009 22804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:08.324707 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.306243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34297 (local address 127.0.0.1:45624)
0924 11:46:08.306700 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.306737 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:08.306886 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:08.307369 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:08.307372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:08.307736 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:08.307875 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.307883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.309255 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.309261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.310237 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.310252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.311373 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.311378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.311511 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.312081 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:08.312111 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:08.314263 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:08.317917 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.317922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.317926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.318388 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.318830 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.318835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.318839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.318994 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.319631 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:08.319637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:08.319776 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 11:46:08.319923 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 11:46:08.320195 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386}
I20260924 11:46:08.325065 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.307644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.97.62:41611)
0924 11:46:08.307797 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.307803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.308028 (+ 225us) server_negotiation.cc:407] Connection header received
0924 11:46:08.308356 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.308361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.308426 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.308500 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.310022 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.311198 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.313124 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.313853 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.313984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.318742 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.318751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.318758 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.318800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.321967 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.322823 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.322831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.322836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.323019 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.323498 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.323503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.323506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.323765 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.323952 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.324488 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.324689 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:46:08.325973 22952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:46:08.343070 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.326889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35550)
0924 11:46:08.327371 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.327390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:08.327655 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:08.328703 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:08.328707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:08.329177 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:08.329351 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.329365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.330697 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.330702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.332071 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.332086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.333139 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.333145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.333316 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.334084 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:08.334112 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:08.336716 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:08.340737 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.340741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.340744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.341220 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.341575 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.341579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.341582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.341693 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.342169 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:08.342173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:08.342371 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 11:46:08.342531 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 11:46:08.342803 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383}
I20260924 11:46:08.343919 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.327178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.0.97.60:37253)
0924 11:46:08.328329 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.328335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.328351 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:46:08.328396 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.328400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.328466 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.328560 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.329514 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.330533 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.332253 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.332836 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.332969 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.336892 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.336916 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.336921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.336964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.340617 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.341338 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.341345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.341348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.341474 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.341790 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.341797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.341800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.342053 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.342237 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.343463 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.343657 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018}
I20260924 11:46:08.344710 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.327547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.0.97.61:34297)
0924 11:46:08.327852 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.327856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.327869 (+ 13us) server_negotiation.cc:407] Connection header received
0924 11:46:08.327903 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.327907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.327963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.328067 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.329584 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.330569 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.332280 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.332918 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.332996 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.338264 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.338271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.338277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.338309 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.341413 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.342402 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.342407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.342411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.342568 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.343077 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.343086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.343091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.343435 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.343691 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.344064 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.344299 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260924 11:46:08.349090 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:46:08.350780 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: Bootstrap starting.
I20260924 11:46:08.351387 22779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "47adb81c82a94f6abe683a0a0529df4c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796
I20260924 11:46:08.351704 22779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 11:46:08.351887 22779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:08.354087 22779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.355223 22952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:08.356019 22779 leader_election.cc:288] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [CANDIDATE]: Term 1 election: Requested vote from peers 328cdb7e8c794a72af5a6d90c25dfd17 (127.0.97.61:34297), cfd016c43f964122a123ea4421101202 (127.0.97.60:37253)
I20260924 11:46:08.356750 22952 log.cc:824] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:08.360567 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: No bootstrap required, opened a new log
I20260924 11:46:08.374542 22952 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.374979 22952 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd016c43f964122a123ea4421101202, State: Initialized, Role: FOLLOWER
I20260924 11:46:08.375563 22952 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
W20260924 11:46:08.382560 22974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:46:08.382683 22974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:46:08.370155 22737 master_runner.cc:186] Error getting master registration for 127.0.97.61:34297: OK, instance_id { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" instance_seqno: 1790250367776022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:46:08.383833 22737 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:46:08.393113 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.393597 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:08.394737 22952 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:08.394861 22927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47adb81c82a94f6abe683a0a0529df4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cfd016c43f964122a123ea4421101202"
I20260924 11:46:08.395367 22927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:08.398959 22927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47adb81c82a94f6abe683a0a0529df4c in term 1.
I20260924 11:46:08.399641 22754 leader_election.cc:302] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47adb81c82a94f6abe683a0a0529df4c, cfd016c43f964122a123ea4421101202; no voters:
I20260924 11:46:08.400107 22959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:46:08.414940 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.387334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.0.97.60:37253)
0924 11:46:08.387539 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.387548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.392658 (+ 5110us) server_negotiation.cc:407] Connection header received
0924 11:46:08.393990 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.393997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.394081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.394165 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.396062 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.397244 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.399289 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.399942 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.400043 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.407262 (+ 7219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.407270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.407274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.407313 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.411259 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.412706 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.412713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.412719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.412870 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.413290 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.413297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.413302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.413657 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.413847 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.414358 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.414578 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:46:08.415984 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.380301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.97.62:41611)
0924 11:46:08.380489 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.380495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.390651 (+ 10156us) server_negotiation.cc:407] Connection header received
0924 11:46:08.391135 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.391140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.391234 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.391303 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.400737 (+ 9434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.401919 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.405139 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.405822 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.405905 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.409616 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.409623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.409630 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.409674 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.413027 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.413712 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.413717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.413719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.413859 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.414268 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.414272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.414275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.414594 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.414832 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.415378 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.415601 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:46:08.415671 22959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [term 1 LEADER]: Becoming Leader. State: Replica: 47adb81c82a94f6abe683a0a0529df4c, State: Running, Role: LEADER
I20260924 11:46:08.417174 22959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.421103 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.386839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.0.97.61:34297)
0924 11:46:08.387035 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.387043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.399180 (+ 12137us) server_negotiation.cc:407] Connection header received
0924 11:46:08.399651 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.399658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.399731 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.399803 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.402640 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.404034 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.406267 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.407085 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.407203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.411522 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.411531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.411537 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.411587 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.417859 (+ 6272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.418696 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.418701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.418705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.418881 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.419312 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.419317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.419321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.419717 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.419947 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.420501 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.420728 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:46:08.422268 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47adb81c82a94f6abe683a0a0529df4c. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.422732 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:08.424240 22984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:46:08.428645 22984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:46:08.441027 22927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [term 1 FOLLOWER]: Refusing update from remote peer 47adb81c82a94f6abe683a0a0529df4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:46:08.442390 22959 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:46:08.457441 22995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:46:08.457718 22995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:46:08.449565 22885 master_runner.cc:186] Error getting master registration for 127.0.97.61:34297: OK, instance_id { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" instance_seqno: 1790250367776022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:46:08.461560 22885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:46:08.481555 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47adb81c82a94f6abe683a0a0529df4c. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.481861 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:08.495467 22878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } attempt: 5
I20260924 11:46:08.497330 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.497627 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:08.498592 22984 catalog_manager.cc:1347] Generated new cluster ID: 4f62ca5a2b254dbf81a7952d63b822aa
I20260924 11:46:08.498679 22984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:46:08.499087 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.499315 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 11:46:08 dist-test-slave-qhrh krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:46:08.517992 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.496645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.0.97.60:37253)
0924 11:46:08.496775 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.496780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.496795 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:46:08.496842 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.496846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.496910 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.497036 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.498161 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.499156 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.500796 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.501518 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.501619 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.505941 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:08.505948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:08.505953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:08.505991 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:08.513722 (+ 7731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.515000 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.515006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.515010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.515203 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:08.515711 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:08.515717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:08.515720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:08.516126 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:08.516330 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.517268 (+ 938us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.517558 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 11:46:08.518896 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.495980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37253 (local address 127.0.0.1:35568)
0924 11:46:08.496345 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:08.496364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:08.496474 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:08.497218 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:08.497227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:08.497753 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:08.497967 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.497979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.499318 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.499325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.500655 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:08.500667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.501779 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.501784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:08.501968 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.502729 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:08.502759 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:08.505670 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:08.513923 (+ 8253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.513929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.513935 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.514845 (+ 910us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.515357 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:08.515363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:08.515366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:08.515566 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:08.516399 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:08.516415 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:08.516590 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 11:46:08.516751 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:46:08.518332 (+ 1581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260924 11:46:08.531872 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: Bootstrap starting.
I20260924 11:46:08.536427 22878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:08.537932 22878 log.cc:824] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:08.538705 22984 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:46:08.540601 22984 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:46:08.551087 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: No bootstrap required, opened a new log
I20260924 11:46:08.560417 22984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c: Generated new TSK 0
I20260924 11:46:08.561503 22984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:46:08.576089 22878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.576684 22878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328cdb7e8c794a72af5a6d90c25dfd17, State: Initialized, Role: FOLLOWER
I20260924 11:46:08.577418 22878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } }
I20260924 11:46:08.591643 22852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:08.591523 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47adb81c82a94f6abe683a0a0529df4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "328cdb7e8c794a72af5a6d90c25dfd17"
I20260924 11:46:08.595176 22852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [term 1 FOLLOWER]: Refusing update from remote peer 47adb81c82a94f6abe683a0a0529df4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:46:08.596236 22878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:08.598814 22853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47adb81c82a94f6abe683a0a0529df4c in term 1.
I20260924 11:46:08.599715 22959 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:46:08.599900 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.600828 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:08.650333 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.651087 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47adb81c82a94f6abe683a0a0529df4c. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:08.651484 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:08.651734 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:08.694725 23020 catalog_manager.cc:1259] Loaded cluster ID: 4f62ca5a2b254dbf81a7952d63b822aa
I20260924 11:46:08.694826 23020 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: loading cluster ID for follower catalog manager: success
I20260924 11:46:08.698901 23020 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: acquiring CA information for follower catalog manager: success
I20260924 11:46:08.701272 23020 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 328cdb7e8c794a72af5a6d90c25dfd17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:46:08.721810 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:08.709200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.0.97.62:41611)
0924 11:46:08.709447 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.709454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.719942 (+ 10488us) server_negotiation.cc:407] Connection header received
0924 11:46:08.720200 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.720208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.720304 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.720421 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.721534 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60818: BlockingRecv error: recv got EOF from 127.0.0.1:60818 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 11:46:08.723569 22882 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:08.716121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.97.61:34297)
0924 11:46:08.716341 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.716348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.721812 (+ 5464us) server_negotiation.cc:407] Connection header received
0924 11:46:08.722082 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.722091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.722206 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.722286 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.723310 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45644: BlockingRecv error: recv got EOF from 127.0.0.1:45644 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260924 11:46:08.725845 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:08.716285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.0.97.60:37253)
0924 11:46:08.716575 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.716583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.724094 (+ 7511us) server_negotiation.cc:407] Connection header received
0924 11:46:08.724367 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.724375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.724470 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.724548 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.725538 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35582: BlockingRecv error: recv got EOF from 127.0.0.1:35582 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 11:46:08.783864 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.769583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.97.61:34297)
0924 11:46:08.769791 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.769798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.776885 (+ 7087us) server_negotiation.cc:407] Connection header received
0924 11:46:08.777166 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.777175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.777412 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.777596 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:08.778120 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.779185 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.781066 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.781668 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.781761 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.782489 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:08.783115 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:08.783218 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.783489 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.783570 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 11:46:08.843465 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:08.831384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.97.60:37253)
0924 11:46:08.831616 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.831626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.841571 (+ 9945us) server_negotiation.cc:407] Connection header received
0924 11:46:08.841895 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.841900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.842052 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.842128 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.843219 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35594: BlockingRecv error: recv got EOF from 127.0.0.1:35594 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:46:08.891198 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.877103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.0.97.61:34297)
0924 11:46:08.877295 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.877302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.884640 (+ 7338us) server_negotiation.cc:407] Connection header received
0924 11:46:08.885102 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.885110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.885211 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.885278 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:08.885921 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.887060 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.889076 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.889716 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.889809 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.890382 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:08.890554 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:08.890612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.890821 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.890901 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 11:46:08.936043 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:08.924659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.0.97.60:37253)
0924 11:46:08.924837 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.924844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.933878 (+ 9034us) server_negotiation.cc:407] Connection header received
0924 11:46:08.934372 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.934382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.934480 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.934579 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.935714 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35598: BlockingRecv error: recv got EOF from 127.0.0.1:35598 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:46:08.984527 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:08.970421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.0.97.61:34297)
0924 11:46:08.970611 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.970617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.977844 (+ 7227us) server_negotiation.cc:407] Connection header received
0924 11:46:08.978337 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.978344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.978431 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.978514 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:08.979186 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.980332 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.982287 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:08.982970 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:08.983065 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:08.983699 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:08.983838 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:08.983891 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:08.984113 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:46:08.984213 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 11:46:08.995679 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:08.993493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.0.97.60:37253)
0924 11:46:08.993651 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:08.993657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:08.994124 (+ 467us) server_negotiation.cc:407] Connection header received
0924 11:46:08.994364 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:08.994373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:08.994466 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:08.994530 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:08.995449 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35606: BlockingRecv error: recv got EOF from 127.0.0.1:35606 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:46:09.010430 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.003443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.0.97.61:34297)
0924 11:46:09.003657 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.003663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.004078 (+ 415us) server_negotiation.cc:407] Connection header received
0924 11:46:09.004368 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.004375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.004466 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.004523 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.005075 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.006255 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.008304 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.008995 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.009084 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.009653 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.009772 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.009834 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.010050 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.010127 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260924 11:46:09.026533 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:09.024219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.0.97.60:37253)
0924 11:46:09.024450 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.024455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.024932 (+ 477us) server_negotiation.cc:407] Connection header received
0924 11:46:09.025185 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.025191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.025298 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.025360 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:09.026275 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35614: BlockingRecv error: recv got EOF from 127.0.0.1:35614 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260924 11:46:09.045132 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.038361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.97.61:34297)
0924 11:46:09.038545 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.038569 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.039001 (+ 432us) server_negotiation.cc:407] Connection header received
0924 11:46:09.039265 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.039271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.039369 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.039483 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.039889 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.041070 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.042951 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.043700 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.043841 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.044285 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.044447 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.044557 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.044701 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.044790 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 11:46:09.061997 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:09.059894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.97.60:37253)
0924 11:46:09.060037 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.060042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.060547 (+ 505us) server_negotiation.cc:407] Connection header received
0924 11:46:09.060705 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.060711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.060819 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.060934 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:09.061754 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35624: BlockingRecv error: recv got EOF from 127.0.0.1:35624 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:46:09.088913 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.080953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.0.97.61:34297)
0924 11:46:09.081094 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.081099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.081525 (+ 426us) server_negotiation.cc:407] Connection header received
0924 11:46:09.081694 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.081702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.081792 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.081905 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.082415 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.083671 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.086762 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.087470 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.087618 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.088201 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.088321 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.088414 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.088580 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.088663 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 11:46:09.104094 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:09.101817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.0.97.60:37253)
0924 11:46:09.102037 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.102043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.102521 (+ 478us) server_negotiation.cc:407] Connection header received
0924 11:46:09.102680 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.102684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.102755 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.102843 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:09.103771 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35626: BlockingRecv error: recv got EOF from 127.0.0.1:35626 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:46:09.145418 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.139538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.97.61:34297)
0924 11:46:09.139753 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.139760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.140159 (+ 399us) server_negotiation.cc:407] Connection header received
0924 11:46:09.140387 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.140395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.140504 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.140613 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.141015 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.142038 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.143435 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.144090 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.144264 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.144621 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.144752 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.144835 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.144979 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.145060 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 11:46:09.153748 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:09.151715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.0.97.60:37253)
0924 11:46:09.151939 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.151947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.152412 (+ 465us) server_negotiation.cc:407] Connection header received
0924 11:46:09.152608 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.152613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.152690 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.152791 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:09.153505 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35638: BlockingRecv error: recv got EOF from 127.0.0.1:35638 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:46:09.207856 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47adb81c82a94f6abe683a0a0529df4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47adb81c82a94f6abe683a0a0529df4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41611 } } peers { permanent_uuid: "328cdb7e8c794a72af5a6d90c25dfd17" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34297 } } peers { permanent_uuid: "cfd016c43f964122a123ea4421101202" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37253 } } }
I20260924 11:46:09.208206 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47adb81c82a94f6abe683a0a0529df4c [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:09.227463 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.221384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.0.97.61:34297)
0924 11:46:09.221633 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.221640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.221985 (+ 345us) server_negotiation.cc:407] Connection header received
0924 11:46:09.222634 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.222641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.222719 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.222807 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.223269 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.224079 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.225614 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.226108 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.226255 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.226669 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.226773 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.226881 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.227067 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.227151 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260924 11:46:09.236352 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:09.234351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.0.97.60:37253)
0924 11:46:09.234552 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.234557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.234817 (+ 260us) server_negotiation.cc:407] Connection header received
0924 11:46:09.235007 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.235011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.235080 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.235160 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:09.236093 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35648: BlockingRecv error: recv got EOF from 127.0.0.1:35648 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:46:09.372442 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.367525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.0.97.61:34297)
0924 11:46:09.367771 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.367778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.368000 (+ 222us) server_negotiation.cc:407] Connection header received
0924 11:46:09.368288 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.368295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.368378 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.368473 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.368881 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.369596 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.370857 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.371271 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.371411 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.371781 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.371873 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.371963 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.372106 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.372198 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 11:46:09.379122 22956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:09.377537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.0.97.60:37253)
0924 11:46:09.377760 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.377766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.378000 (+ 234us) server_negotiation.cc:407] Connection header received
0924 11:46:09.378123 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.378126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.378193 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.378293 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:09.378920 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35662: BlockingRecv error: recv got EOF from 127.0.0.1:35662 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:46:09.459741 22995 catalog_manager.cc:1259] Loaded cluster ID: 4f62ca5a2b254dbf81a7952d63b822aa
I20260924 11:46:09.459872 22995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: loading cluster ID for follower catalog manager: success
I20260924 11:46:09.462522 22995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: acquiring CA information for follower catalog manager: success
I20260924 11:46:09.464556 22995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfd016c43f964122a123ea4421101202: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:46:09.643265 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.638446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.0.97.61:34297)
0924 11:46:09.638687 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.638694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.638832 (+ 138us) server_negotiation.cc:407] Connection header received
0924 11:46:09.639044 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.639050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.639117 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.639207 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.639581 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.640385 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.641679 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.642068 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.642202 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.642600 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.642695 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.642789 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.642932 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.643006 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:46:09.654078 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.649359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.97.60:37253)
0924 11:46:09.649545 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.649549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.649729 (+ 180us) server_negotiation.cc:407] Connection header received
0924 11:46:09.649838 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.649842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.649991 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.650109 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.650461 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.651117 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.652259 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.652630 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.652723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.653119 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.653620 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.653740 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.653781 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.653862 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 11:46:09.668035 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.659893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.0.97.62:41611)
0924 11:46:09.660401 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.660407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.661075 (+ 668us) server_negotiation.cc:407] Connection header received
0924 11:46:09.661409 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.661415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.661634 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.661778 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.662819 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.663631 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.665829 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.666212 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.666322 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.666805 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.667300 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.667408 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.667556 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.667641 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":283,"threads_started":1}
I20260924 11:46:09.668263 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.659965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.0.97.61:34297)
0924 11:46:09.660204 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.660211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.661778 (+ 1567us) server_negotiation.cc:407] Connection header received
0924 11:46:09.661944 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.661950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.662039 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.662127 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.663037 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.664022 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.666451 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.666921 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.667090 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.667464 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.667592 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.667670 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.667856 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.667931 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 11:46:09.668903 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:09.660132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.0.97.60:37253)
0924 11:46:09.660345 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:09.660350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:09.661575 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 11:46:09.661902 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:09.661907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:09.661982 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:09.662079 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:09.662855 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.663880 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.666446 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:09.667006 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:09.667143 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:09.667956 (+ 813us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:09.668096 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:09.668193 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:09.668449 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 11:46:09.668521 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:46:09.673532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22737
I20260924 11:46:09.674388 22799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:09.776882 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22737
W20260924 11:46:09.796906 22830 connection.cc:535] server connection from 127.0.0.1:45614 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:46:09.797735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22811
I20260924 11:46:09.798767 22873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:09.898212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22811
I20260924 11:46:09.917752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 22885
I20260924 11:46:09.918903 22947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:10.020506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 22885
2026-09-24T11:46:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4362 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:46:10 dist-test-slave-qhrh krb5kdc[23145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:46:10 dist-test-slave-qhrh krb5kdc[23145](info): set up 2 sockets
Sep 24 11:46:10 dist-test-slave-qhrh krb5kdc[23145](info): commencing operation
krb5kdc: starting...
W20260924 11:46:11.076692 388 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/test-user.keytab.
Sep 24 11:46:11 dist-test-slave-qhrh krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:46:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 11:46:11.303480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43585,127.0.97.61:37167,127.0.97.60:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:46:11.611717 23158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:46:11.612098 23158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:11.612243 23158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:11.612375 23158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:11.619349 23158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:11.619465 23158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:11.619565 23158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:11.619657 23158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:11.619742 23158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:11.629277 23158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43585,127.0.97.61:37167,127.0.97.60:43623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:11.630573 23158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:11.631843 23158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:46:11.643052 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:11.643103 23166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:11.643335 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:11.643651 23158 server_base.cc:1034] running on GCE node
I20260924 11:46:11.644410 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:11.645651 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:11.646849 23158 hybrid_clock.cc:648] HybridClock initialized: now 1790250371646784 us; error 39 us; skew 500 ppm
I20260924 11:46:11.647153 23158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:11 dist-test-slave-qhrh krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:11.650148 23158 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 11:46:11.651423 23158 webserver.cc:460] Webserver started at http://127.0.97.62:35335/ using document root <none> and password file <none>
I20260924 11:46:11.652060 23158 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:11.652181 23158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:11.652495 23158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:11.655135 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data/instance:
uuid: "5748f7706233450c993531fa75fa2ac9"
format_stamp: "Formatted at 2026-09-24 11:46:11 on dist-test-slave-qhrh"
server_key: "d3b535a303a7c2cfa7762fcbea2bc2f6"
server_key_iv: "fc0970703a96d7ca2a214374a4c0e423"
server_key_version: "encryptionkey@0"
I20260924 11:46:11.655825 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal/instance:
uuid: "5748f7706233450c993531fa75fa2ac9"
format_stamp: "Formatted at 2026-09-24 11:46:11 on dist-test-slave-qhrh"
server_key: "d3b535a303a7c2cfa7762fcbea2bc2f6"
server_key_iv: "fc0970703a96d7ca2a214374a4c0e423"
server_key_version: "encryptionkey@0"
I20260924 11:46:11.660653 23158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260924 11:46:11.663710 23173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:11.664991 23158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:46:11.665169 23158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal
uuid: "5748f7706233450c993531fa75fa2ac9"
format_stamp: "Formatted at 2026-09-24 11:46:11 on dist-test-slave-qhrh"
server_key: "d3b535a303a7c2cfa7762fcbea2bc2f6"
server_key_iv: "fc0970703a96d7ca2a214374a4c0e423"
server_key_version: "encryptionkey@0"
I20260924 11:46:11.665374 23158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:11.677412 23158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:11.680912 23158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:11.681147 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:11.697642 23158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43585
I20260924 11:46:11.697665 23224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43585 every 8 connection(s)
I20260924 11:46:11.698899 23158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/data/info.pb
I20260924 11:46:11.702248 23225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:11.704715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 23158
I20260924 11:46:11.704880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-0/wal/instance
I20260924 11:46:11.705265 388 external_mini_cluster.cc:1428] Setting key f99f1f89298de8e58d5c05e1c001e8dc
I20260924 11:46:11.708590 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:11 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260924 11:46:11.731381 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.711387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.0.97.62:43585)
0924 11:46:11.712207 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:11.712220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:11.712641 (+ 421us) server_negotiation.cc:407] Connection header received
0924 11:46:11.714457 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:11.714467 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:11.714745 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:11.714945 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:11.715977 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:11.717794 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:11.719216 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:11.720117 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:11.720336 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:11.723453 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:11.723466 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:11.723482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:11.723522 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:11.727378 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:11.728034 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:11.728056 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:11.728059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:11.728267 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:11.728660 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:11.728666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:11.728670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:11.729341 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:11.729575 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:11.730080 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 11:46:11.730454 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":555,"thread_start_us":341,"threads_started":1}
I20260924 11:46:11.731400 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.711380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43585 (local address 127.0.0.1:44720)
0924 11:46:11.712461 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.712502 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:11.714103 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:11.714959 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:11.714964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:11.715462 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:11.715781 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:11.715796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:11.717937 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:11.717940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:11.719078 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:11.719089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:11.720358 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:11.720363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:11.720473 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:11.721116 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:11.721140 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:11.723190 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:11.727505 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:11.727514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:11.727523 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:11.727894 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:11.728379 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:11.728382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:11.728385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:11.728547 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:11.729490 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:11.729495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:11.729700 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 11:46:11.729896 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 11:46:11.730284 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"spinlock_wait_cycles":18688,"thread_start_us":627,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260924 11:46:11.734249 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:11.735183 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.734853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46886)
0924 11:46:11.734996 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.735089 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:46:11.735394 23175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
I20260924 11:46:11.736838 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.736506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46888)
0924 11:46:11.736650 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.736719 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:46:11.737013 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 11:46:11.760594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37167
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43585,127.0.97.61:37167,127.0.97.60:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:46:11.786427 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } attempt: 1
I20260924 11:46:11.787655 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.787197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46902)
0924 11:46:11.787421 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.787522 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 11:46:11.788609 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.788302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46916)
0924 11:46:11.788415 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.788486 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 11:46:11.788839 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
I20260924 11:46:11.857070 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } attempt: 2
I20260924 11:46:11.858384 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.857807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46932)
0924 11:46:11.858035 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.858155 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:46:11.859566 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.859205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46944)
0924 11:46:11.859384 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.859456 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:46:11.859722 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
I20260924 11:46:11.951048 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } attempt: 3
I20260924 11:46:11.952387 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.951716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46958)
0924 11:46:11.951991 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.952182 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 11:46:11.953521 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:11.953045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46960)
0924 11:46:11.953226 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:11.953335 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 11:46:11.953727 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
W20260924 11:46:12.067325 23232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:46:12.067668 23232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:12.067811 23232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:12.067931 23232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:12.075176 23232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:12.075269 23232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:12.075361 23232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:12.075434 23232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:12.075513 23232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:12.084544 23232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43585,127.0.97.61:37167,127.0.97.60:43623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:12.085804 23232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:12.087059 23232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:46:12.096971 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } attempt: 4
I20260924 11:46:12.098423 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.097713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46970)
0924 11:46:12.098132 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.098244 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260924 11:46:12.099866 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.099414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46986)
0924 11:46:12.099591 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.099695 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 11:46:12.099855 23240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:12.100104 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37167: connect: Connection refused (error 111)
W20260924 11:46:12.100507 23238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:12.100816 23237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:12.101051 23232 server_base.cc:1034] running on GCE node
I20260924 11:46:12.101810 23232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:12.103133 23232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:12.104370 23232 hybrid_clock.cc:648] HybridClock initialized: now 1790250372104291 us; error 60 us; skew 500 ppm
I20260924 11:46:12.104676 23232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:12.107892 23232 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 11:46:12.109361 23232 webserver.cc:460] Webserver started at http://127.0.97.61:41987/ using document root <none> and password file <none>
I20260924 11:46:12.110019 23232 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:12.110145 23232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:12.110440 23232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:12.113220 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data/instance:
uuid: "313e4eee942c424a9b0953d9d56e39d4"
format_stamp: "Formatted at 2026-09-24 11:46:12 on dist-test-slave-qhrh"
server_key: "58615c1440e6015579e5ab59d77cbbe1"
server_key_iv: "111529fc1fc658fe762b3f74b2f1afda"
server_key_version: "encryptionkey@0"
I20260924 11:46:12.113953 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal/instance:
uuid: "313e4eee942c424a9b0953d9d56e39d4"
format_stamp: "Formatted at 2026-09-24 11:46:12 on dist-test-slave-qhrh"
server_key: "58615c1440e6015579e5ab59d77cbbe1"
server_key_iv: "111529fc1fc658fe762b3f74b2f1afda"
server_key_version: "encryptionkey@0"
I20260924 11:46:12.118662 23232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:46:12.121888 23247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:12.123174 23232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 11:46:12.123351 23232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal
uuid: "313e4eee942c424a9b0953d9d56e39d4"
format_stamp: "Formatted at 2026-09-24 11:46:12 on dist-test-slave-qhrh"
server_key: "58615c1440e6015579e5ab59d77cbbe1"
server_key_iv: "111529fc1fc658fe762b3f74b2f1afda"
server_key_version: "encryptionkey@0"
I20260924 11:46:12.123572 23232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:12.151178 23232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:12.155570 23232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:12.155831 23232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:12.173532 23232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37167
I20260924 11:46:12.173563 23298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37167 every 8 connection(s)
I20260924 11:46:12.174793 23232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/data/info.pb
I20260924 11:46:12.178186 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:12.183271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 23232
I20260924 11:46:12.183427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-1/wal/instance
I20260924 11:46:12.183900 388 external_mini_cluster.cc:1428] Setting key 724b763e6acc2b7f53cf8173fd5691cb
I20260924 11:46:12.184983 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:46:12.206290 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.188214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.0.97.62:43585)
0924 11:46:12.188446 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.188454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.189518 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 11:46:12.191258 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.191267 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.191359 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.191509 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.192899 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.193924 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.196117 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.196541 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.196637 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.200657 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.200661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.200665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.200695 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.203326 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.204014 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.204026 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.204029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.204179 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.204630 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.204635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.204638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.204913 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.205064 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.205742 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.205969 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:46:12.207232 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.188613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43585 (local address 127.0.0.1:44728)
0924 11:46:12.189346 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.189388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.190984 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.191707 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.191714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.192241 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.192733 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.192747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.194075 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.194083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.195979 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.195995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.196758 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.196767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.197071 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.197785 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.197823 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.200444 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.203463 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.203471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.203486 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.203891 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.204301 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.204306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.204310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.204511 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.205035 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.205042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.205376 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.205623 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.206131 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 11:46:12.209064 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:46:12.224100 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.209832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.97.61:37167)
0924 11:46:12.210382 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.210387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.210411 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:46:12.210451 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.210456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.210699 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.210865 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.211713 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.212753 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.214484 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.214995 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.215084 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.218462 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.218484 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.218496 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.218524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.221316 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.222069 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.222075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.222077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.222180 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.222592 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.222595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.222597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.222896 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.223050 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.223469 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.223794 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":392,"thread_start_us":285,"threads_started":1}
I20260924 11:46:12.224123 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.209659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:46992)
0924 11:46:12.209805 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.209820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.209894 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.210881 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.210892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.211323 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.211531 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.211555 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.212905 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.212912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.214356 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.214372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.215144 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.215152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.215387 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.215948 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.215972 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.218266 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.221437 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.221444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.221450 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.221955 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.222312 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.222319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.222326 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.222496 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.223035 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.223048 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.223206 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.223360 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.223661 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 11:46:12.226635 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:12.228018 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.227448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39358)
0924 11:46:12.227641 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.227811 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:46:12.228344 23250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.230104 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.229611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39364)
0924 11:46:12.229821 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.229953 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 11:46:12.230343 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 11:46:12.242937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpreJ74/build/asan/bin/kudu
/tmp/dist-test-taskpreJ74/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43623
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43585,127.0.97.61:37167,127.0.97.60:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:46:12.279644 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 1
I20260924 11:46:12.280856 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.280313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39370)
0924 11:46:12.280560 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.280668 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 11:46:12.281852 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.281492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39378)
0924 11:46:12.281630 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.281732 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 11:46:12.282135 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.350365 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 2
I20260924 11:46:12.351496 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.350996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39386)
0924 11:46:12.351213 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.351318 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:46:12.352579 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.352229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39396)
0924 11:46:12.352363 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.352444 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 11:46:12.352789 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.399401 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } attempt: 5
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:46:12.415535 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.400509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.97.61:37167)
0924 11:46:12.400692 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.400697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.400717 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:46:12.400782 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.400787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.400877 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.402039 (+ 1162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.402187 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.403272 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.404970 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.405614 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.405733 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.409651 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.409657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.409663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.409707 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.412988 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.413672 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.413677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.413680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.413843 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.414243 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.414248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.414251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.414511 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.414654 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.415067 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.415260 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:46:12.415535 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.400173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:47002)
0924 11:46:12.400388 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.400423 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.400551 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.401099 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.401104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.401616 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.401864 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.401876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.403442 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.403447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.404785 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.404800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.405760 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.405766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.405979 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.406639 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.406666 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.409419 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.413118 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.413124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.413128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.413559 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.413972 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.413977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.413979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.414141 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.414623 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.414628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.414785 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.414938 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.415194 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 11:46:12.417106 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:12.418267 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.417831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39400)
0924 11:46:12.417991 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.418078 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:46:12.419466 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.419000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39410)
0924 11:46:12.419230 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.419336 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 11:46:12.419679 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.444027 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 3
I20260924 11:46:12.445235 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.444721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39424)
0924 11:46:12.444941 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.445050 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 11:46:12.446225 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.445801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39434)
0924 11:46:12.446018 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.446099 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260924 11:46:12.446439 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.470897 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 1
I20260924 11:46:12.471906 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.471477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39442)
0924 11:46:12.471664 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.471757 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:46:12.472908 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.472622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39446)
0924 11:46:12.472760 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.472818 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:46:12.473114 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.541412 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 2
I20260924 11:46:12.542462 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.542018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39450)
0924 11:46:12.542202 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.542325 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:46:12.543429 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.543108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39466)
0924 11:46:12.543261 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.543319 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:46:12.543622 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
W20260924 11:46:12.570250 23306 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:46:12.570595 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:46:12.570716 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:46:12.570852 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:46:12.577520 23306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:46:12.577630 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:46:12.577752 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:46:12.577837 23306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:46:12.577922 23306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:46:12.586408 23306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43585,127.0.97.61:37167,127.0.97.60:43623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:46:12.587610 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:46:12.588868 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:46:12.589646 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 4
I20260924 11:46:12.590723 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.590219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39472)
0924 11:46:12.590444 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.590552 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:46:12.591636 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.591317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39476)
0924 11:46:12.591452 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.591522 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 11:46:12.591826 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
W20260924 11:46:12.600060 23314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:12.600096 23312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:46:12.600117 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:46:12.600987 23306 server_base.cc:1034] running on GCE node
I20260924 11:46:12.601698 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:46:12.602890 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:46:12.604184 23306 hybrid_clock.cc:648] HybridClock initialized: now 1790250372604077 us; error 50 us; skew 500 ppm
I20260924 11:46:12.604445 23306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:46:12.607210 23306 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 11:46:12.608613 23306 webserver.cc:460] Webserver started at http://127.0.97.60:43017/ using document root <none> and password file <none>
I20260924 11:46:12.609186 23306 fs_manager.cc:359] Metadata directory not provided
I20260924 11:46:12.609263 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:46:12.609488 23306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:46:12.612128 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data/instance:
uuid: "bc1e272e3ffc499788d0074a4a4943ef"
format_stamp: "Formatted at 2026-09-24 11:46:12 on dist-test-slave-qhrh"
server_key: "39aef5a7fc8faaa520b38e69d8f6574b"
server_key_iv: "b2b79649b97ec914516e0009ec802854"
server_key_version: "encryptionkey@0"
I20260924 11:46:12.612874 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal/instance:
uuid: "bc1e272e3ffc499788d0074a4a4943ef"
format_stamp: "Formatted at 2026-09-24 11:46:12 on dist-test-slave-qhrh"
server_key: "39aef5a7fc8faaa520b38e69d8f6574b"
server_key_iv: "b2b79649b97ec914516e0009ec802854"
server_key_version: "encryptionkey@0"
I20260924 11:46:12.617303 23306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 11:46:12.620335 23321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:46:12.621574 23306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:46:12.621698 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal
uuid: "bc1e272e3ffc499788d0074a4a4943ef"
format_stamp: "Formatted at 2026-09-24 11:46:12 on dist-test-slave-qhrh"
server_key: "39aef5a7fc8faaa520b38e69d8f6574b"
server_key_iv: "b2b79649b97ec914516e0009ec802854"
server_key_version: "encryptionkey@0"
I20260924 11:46:12.621853 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:46:12.633575 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:46:12.637528 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:46:12.637722 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:46:12.649791 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 3
I20260924 11:46:12.650811 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.650335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39490)
0924 11:46:12.650554 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.650628 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:46:12.651804 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.651443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39494)
0924 11:46:12.651653 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.651716 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 11:46:12.652004 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43623: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43623: connect: Connection refused (error 111)
I20260924 11:46:12.655196 23306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43623
I20260924 11:46:12.655213 23372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43623 every 8 connection(s)
I20260924 11:46:12.656440 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/data/info.pb
I20260924 11:46:12.659734 23373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:46:12.665226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpreJ74/build/asan/bin/kudu as pid 23306
I20260924 11:46:12.665405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpreJ74/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250027667721-388-0/minicluster-data/master-2/wal/instance
I20260924 11:46:12.665778 388 external_mini_cluster.cc:1428] Setting key 1384df8dd6a5808f0a99a443f2dc7d61
I20260924 11:46:12.665949 23373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:46:12.684046 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.668392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.0.97.62:43585)
0924 11:46:12.668593 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.668597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.669343 (+ 746us) server_negotiation.cc:407] Connection header received
0924 11:46:12.670379 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.670384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.670460 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.670600 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.671760 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.672478 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.673998 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.674549 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.674640 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.678031 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.678037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.678042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.678090 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.680935 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.681703 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.681706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.681708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.681807 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.682239 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.682243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.682245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.682609 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.682755 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.683506 (+ 751us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.683710 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 11:46:12.685251 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.668486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43585 (local address 127.0.0.1:44730)
0924 11:46:12.669217 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.669249 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.670163 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.670678 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.670684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.671290 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.671593 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.671607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.672619 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.672624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.673861 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.673875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.674742 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.674749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.674949 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.675675 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.675697 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.677846 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.681040 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.681048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.681057 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.681594 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.681905 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.681909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.681913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.682104 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.682723 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.682728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.683060 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.683367 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.683703 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":392,"threads_started":1}
I20260924 11:46:12.685541 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.667210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.0.97.60:43623)
0924 11:46:12.667955 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.667974 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.668019 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:46:12.670154 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.670172 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.670380 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.670564 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.671799 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.673158 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.674364 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.674963 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.675073 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.679148 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.679156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.679165 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.679192 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.682133 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.682905 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.682911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.682914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.683060 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.683468 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.683472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.683475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.683842 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.684123 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.684907 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.685111 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":354,"threads_started":1}
I20260924 11:46:12.687404 23373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 11:46:12.700393 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.687994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37167 (local address 127.0.0.1:47004)
0924 11:46:12.688124 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.688162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.688249 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.688663 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.688668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.689081 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.689268 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.689280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.690309 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.690313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.691380 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.691393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.692015 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.692019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.692193 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.692702 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.692725 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.694881 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.698075 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.698082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.698086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.698522 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.698891 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.698894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.698896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.699026 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.699543 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.699547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.699676 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.699828 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.700053 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 11:46:12.700501 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.688127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.97.61:37167)
0924 11:46:12.688343 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.688349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.688367 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:46:12.688427 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.688432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.688511 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.688619 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.689404 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.690177 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.691507 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.691889 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.691995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.695050 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.695062 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.695065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.695094 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.697939 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.698635 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.698641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.698645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.698787 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.699153 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.699159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.699164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.699422 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.699574 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.699932 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.700122 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 11:46:12.701196 23373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:46:12.703419 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.688239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.0.97.62:43585)
0924 11:46:12.688376 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.688382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.688397 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:46:12.688438 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.688442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.688509 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.688617 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.689810 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.690621 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.691897 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.692460 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.692577 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.696627 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.696631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.696634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.696658 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.700414 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.701418 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.701423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.701425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.701571 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.701954 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.701958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.701964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.702348 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.702641 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.702915 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.703123 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 11:46:12.715883 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.701821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39514)
0924 11:46:12.701978 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.701995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.702103 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.702640 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.702645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.703085 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.703381 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.703392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.704674 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.704679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.706307 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.706321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.707146 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.707151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.707368 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.707951 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.707973 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.709832 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.713096 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.713100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.713104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.713566 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.714422 (+ 856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.714427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.714432 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.714605 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.715164 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.715167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.715267 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.715399 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.715647 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 11:46:12.716116 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.702127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.0.97.60:43623)
0924 11:46:12.702272 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.702277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.702311 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:46:12.702356 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.702361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.702421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.702508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.703653 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.704552 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.706443 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.706998 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.707118 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.710048 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.710055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.710059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.710093 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.712970 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.713665 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.713668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.713670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.713766 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.714731 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.714736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.714739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.715062 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.715190 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.715521 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.715785 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":43}
I20260924 11:46:12.719381 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.704979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.0.97.61:37167)
0924 11:46:12.705120 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.705125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.705184 (+ 59us) server_negotiation.cc:407] Connection header received
0924 11:46:12.705351 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.705356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.705419 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.705513 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.706577 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.707486 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.708875 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.709225 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.709338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.713932 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.713936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.713939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.713964 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.716921 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.717609 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.717616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.717622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.717765 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.718120 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.718125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.718127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.718427 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.718692 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.718861 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.719077 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:46:12.721608 23373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: Bootstrap starting.
I20260924 11:46:12.723699 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:46:12.724869 23373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:12.725951 23373 log.cc:824] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:12.728593 23373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: No bootstrap required, opened a new log
I20260924 11:46:12.726027 23200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5748f7706233450c993531fa75fa2ac9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44740
I20260924 11:46:12.734889 23373 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.735191 23373 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1e272e3ffc499788d0074a4a4943ef, State: Initialized, Role: FOLLOWER
I20260924 11:46:12.735594 23373 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.736459 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:12.736811 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:12.737320 23373 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:46:12.745951 23390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:46:12.746088 23390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:46:12.743913 23306 master_runner.cc:186] Error getting master registration for 127.0.97.62:43585: OK, instance_id { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" instance_seqno: 1790250371692821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:46:12.754643 23306 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:46:12.829224 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 4
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:46:12.845091 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.830473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.0.97.60:43623)
0924 11:46:12.830708 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.830726 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.830749 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:46:12.830822 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.830828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.830907 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.831018 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.831911 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.832950 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.834723 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.835401 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.835509 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.839302 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.839309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.839313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.839345 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.842388 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.843143 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.843147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.843150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.843261 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.843648 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.843652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.843655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.844043 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.844204 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.844573 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.844772 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:46:12.845103 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.830022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39518)
0924 11:46:12.830266 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.830287 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.830416 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.831083 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.831089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.831578 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.831769 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.831781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.833091 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.833097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.834556 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.834572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.835541 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.835547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.835736 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.836360 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.836382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.839127 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.842539 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.842545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.842550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.843039 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.843353 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.843357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.843361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.843530 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.844182 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.844187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.844347 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.844484 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.844747 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:46:12.849942 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9: Bootstrap starting.
I20260924 11:46:12.852968 23225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:12.853978 23225 log.cc:824] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:46:12.856592 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9: No bootstrap required, opened a new log
I20260924 11:46:12.862202 23225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.862670 23225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5748f7706233450c993531fa75fa2ac9, State: Initialized, Role: FOLLOWER
I20260924 11:46:12.863152 23225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.864105 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:12.864517 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:12.865195 23225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:12.865324 23200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 11:46:12.865520 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:12.868825 23200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.870469 23200 leader_election.cc:288] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [CANDIDATE]: Term 1 election: Requested vote from peers 313e4eee942c424a9b0953d9d56e39d4 (127.0.97.61:37167), bc1e272e3ffc499788d0074a4a4943ef (127.0.97.60:43623)
I20260924 11:46:12.871066 23348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748f7706233450c993531fa75fa2ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc1e272e3ffc499788d0074a4a4943ef"
I20260924 11:46:12.871536 23348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:12.875840 23348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5748f7706233450c993531fa75fa2ac9 in term 1.
I20260924 11:46:12.876603 23176 leader_election.cc:302] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5748f7706233450c993531fa75fa2ac9, bc1e272e3ffc499788d0074a4a4943ef; no voters:
I20260924 11:46:12.877053 23393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:46:12.888666 23393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 5748f7706233450c993531fa75fa2ac9, State: Running, Role: LEADER
W20260924 11:46:12.889537 23327 tablet.cc:2369] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:46:12.889596 23393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.890982 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } attempt: 5
Sep 24 11:46:12 dist-test-slave-qhrh krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 11:46:12.905436 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5748f7706233450c993531fa75fa2ac9. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:12.905905 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:12.912107 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.891561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43623 (local address 127.0.0.1:39524)
0924 11:46:12.891945 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 11:46:12.891963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:46:12.892078 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:46:12.892720 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:46:12.892727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:46:12.893407 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:46:12.893665 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.893682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.895469 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.895477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.897171 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:46:12.897185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.898282 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.898288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:46:12.898517 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.900031 (+ 1514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:46:12.900060 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:46:12.903264 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:46:12.908051 (+ 4787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.908057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.908061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.908720 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.909552 (+ 832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:46:12.909569 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:46:12.909575 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:46:12.909759 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:46:12.910839 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:46:12.910846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:46:12.911036 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 11:46:12.911222 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 11:46:12.911587 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260924 11:46:12.913618 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.891942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.0.97.60:43623)
0924 11:46:12.892142 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.892147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.892197 (+ 50us) server_negotiation.cc:407] Connection header received
0924 11:46:12.892268 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.892275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.892374 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.892478 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.893849 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.895282 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.897336 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.897951 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.898082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.903495 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.903505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.903512 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.903577 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.907844 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.908960 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.908967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.908972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.909138 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.909968 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.909976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.909981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.910366 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.910518 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.912863 (+ 2345us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.913122 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:46:12.916680 23408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:46:12.923148 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: Bootstrap starting.
I20260924 11:46:12.930094 23299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: Neither blocks nor log segments found. Creating new log.
I20260924 11:46:12.930874 23408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:46:12.932092 23299 log.cc:824] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: Log is configured to *not* fsync() on all Append() calls
W20260924 11:46:12.933657 23179 tablet.cc:2369] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:46:12.936769 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.899149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.97.62:43585)
0924 11:46:12.899373 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.899380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.916331 (+ 16951us) server_negotiation.cc:407] Connection header received
0924 11:46:12.917282 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.917287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.917363 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.917503 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.920728 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.922072 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.924400 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.925245 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.925358 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.929324 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.929333 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.929339 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.929390 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.933428 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.934286 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.934292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.934295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.934454 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.934854 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.934859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.934862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.935245 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.935501 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.936161 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.936404 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:46:12.944962 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: No bootstrap required, opened a new log
I20260924 11:46:12.952739 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.926712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.97.61:37167)
0924 11:46:12.926907 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.926914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.931787 (+ 4873us) server_negotiation.cc:407] Connection header received
0924 11:46:12.933270 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.933274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.933352 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.933416 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.938005 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.938947 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.940839 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.941632 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.941754 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.945345 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.945354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.945362 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.945409 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.949368 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.950155 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.950162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.950166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.950298 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.950730 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.950737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.950741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.951193 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.951436 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.952033 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.952371 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:46:12.955788 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:12.898358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.0.97.60:43623)
0924 11:46:12.916243 (+ 17885us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:12.916250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:12.926128 (+ 9878us) server_negotiation.cc:407] Connection header received
0924 11:46:12.929031 (+ 2903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:12.929038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:12.929144 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:12.929212 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:12.933857 (+ 4645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.934913 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.941345 (+ 6432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:12.941970 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:12.942088 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:12.947573 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:46:12.947583 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:46:12.947591 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:46:12.947633 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:46:12.951002 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.952419 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.952424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.952427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.952554 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:46:12.953916 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:46:12.953922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:46:12.953928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:46:12.954270 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:46:12.954634 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:12.955094 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 11:46:12.955331 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17770,"thread_start_us":22176,"threads_started":1}
I20260924 11:46:12.960247 23299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.960608 23299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313e4eee942c424a9b0953d9d56e39d4, State: Initialized, Role: FOLLOWER
I20260924 11:46:12.961112 23299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } }
I20260924 11:46:12.962023 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:12.962422 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:12.963260 23299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:46:12.963303 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748f7706233450c993531fa75fa2ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "313e4eee942c424a9b0953d9d56e39d4"
I20260924 11:46:12.963761 23274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:46:12.966699 23274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5748f7706233450c993531fa75fa2ac9 in term 1.
I20260924 11:46:12.973953 23348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [term 1 FOLLOWER]: Refusing update from remote peer 5748f7706233450c993531fa75fa2ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:46:12.974107 23274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [term 1 FOLLOWER]: Refusing update from remote peer 5748f7706233450c993531fa75fa2ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:46:12.974987 23401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:46:12.975291 23393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:46:12.997910 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5748f7706233450c993531fa75fa2ac9. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:12.998201 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:13.002540 23416 mvcc.cc:205] Tried to move back new op lower bound from 7332865527617986560 to 7332865527369502720. Current Snapshot: MvccSnapshot[applied={T|T < 7332865527617986560}]
I20260924 11:46:13.034291 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5748f7706233450c993531fa75fa2ac9. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:13.034715 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:13.035276 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:13.035620 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:13.037115 23408 catalog_manager.cc:1347] Generated new cluster ID: 71c94810644a4e978fb6c0cf612870c7
I20260924 11:46:13.037217 23408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:46:13.045020 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:13.045346 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:46:13.048324 23424 mvcc.cc:205] Tried to move back new op lower bound from 7332865527617986560 to 7332865527369502720. Current Snapshot: MvccSnapshot[applied={T|T < 7332865527617986560}]
I20260924 11:46:13.052795 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:13.053077 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:13.056500 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5748f7706233450c993531fa75fa2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748f7706233450c993531fa75fa2ac9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } peers { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37167 } } peers { permanent_uuid: "bc1e272e3ffc499788d0074a4a4943ef" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43623 } } }
I20260924 11:46:13.056728 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:46:13.049975 23158 master_runner.cc:186] Error getting master registration for 127.0.97.61:37167: OK, instance_id { permanent_uuid: "313e4eee942c424a9b0953d9d56e39d4" instance_seqno: 1790250372168511 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:46:13.062011 23158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:46:13.065613 23438 catalog_manager.cc:1259] Loaded cluster ID: 71c94810644a4e978fb6c0cf612870c7
I20260924 11:46:13.065701 23438 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: loading cluster ID for follower catalog manager: success
W20260924 11:46:13.067015 23438 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 11:46:13.067152 23438 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:46:13.082633 23408 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:46:13.084506 23408 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:46:13.105588 23408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5748f7706233450c993531fa75fa2ac9: Generated new TSK 0
I20260924 11:46:13.106745 23408 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:46:13.257609 23227 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.234209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.0.97.62:43585)
0924 11:46:13.234418 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.234427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.255381 (+ 20954us) server_negotiation.cc:407] Connection header received
0924 11:46:13.255933 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.255940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.256068 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.256209 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.257319 (+ 1110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44764: BlockingRecv error: recv got EOF from 127.0.0.1:44764 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 11:46:13.259783 23376 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.241246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.97.60:43623)
0924 11:46:13.241445 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.241455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.258405 (+ 16950us) server_negotiation.cc:407] Connection header received
0924 11:46:13.258570 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.258589 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.258694 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.258788 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.259530 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39538: BlockingRecv error: recv got EOF from 127.0.0.1:39538 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 11:46:13.263180 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.261016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.0.97.61:37167)
0924 11:46:13.261206 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.261212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.261628 (+ 416us) server_negotiation.cc:407] Connection header received
0924 11:46:13.261865 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.261870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.261971 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.262047 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.262953 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47042: BlockingRecv error: recv got EOF from 127.0.0.1:47042 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 11:46:13.313345 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.300350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.97.61:37167)
0924 11:46:13.300526 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.300532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.311179 (+ 10647us) server_negotiation.cc:407] Connection header received
0924 11:46:13.311712 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.311718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.311876 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.311954 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.313056 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47044: BlockingRecv error: recv got EOF from 127.0.0.1:47044 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 11:46:13.362043 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.349297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.0.97.61:37167)
0924 11:46:13.349486 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.349493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.360058 (+ 10565us) server_negotiation.cc:407] Connection header received
0924 11:46:13.360596 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.360604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.360697 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.360777 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.361810 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47054: BlockingRecv error: recv got EOF from 127.0.0.1:47054 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 11:46:13.411494 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.398577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.0.97.61:37167)
0924 11:46:13.398781 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.398790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.409424 (+ 10634us) server_negotiation.cc:407] Connection header received
0924 11:46:13.409994 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.410000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.410119 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.410198 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.411208 (+ 1010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47068: BlockingRecv error: recv got EOF from 127.0.0.1:47068 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 11:46:13.457643 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.450662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.0.97.61:37167)
0924 11:46:13.450889 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.450895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.456179 (+ 5284us) server_negotiation.cc:407] Connection header received
0924 11:46:13.456422 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.456427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.456510 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.456632 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.457362 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47076: BlockingRecv error: recv got EOF from 127.0.0.1:47076 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260924 11:46:13.470770 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.468727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.0.97.61:37167)
0924 11:46:13.468938 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.468949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.469423 (+ 474us) server_negotiation.cc:407] Connection header received
0924 11:46:13.469703 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.469712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.469776 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.469877 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.470524 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47078: BlockingRecv error: recv got EOF from 127.0.0.1:47078 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 11:46:13.492093 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.490082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.0.97.61:37167)
0924 11:46:13.490292 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.490301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.490794 (+ 493us) server_negotiation.cc:407] Connection header received
0924 11:46:13.491000 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.491004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.491086 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.491196 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.491858 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47088: BlockingRecv error: recv got EOF from 127.0.0.1:47088 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 11:46:13.530483 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.527978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.0.97.61:37167)
0924 11:46:13.528219 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.528229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.528922 (+ 693us) server_negotiation.cc:407] Connection header received
0924 11:46:13.529184 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.529190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.529300 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.529402 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.530219 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47090: BlockingRecv error: recv got EOF from 127.0.0.1:47090 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 11:46:13.600955 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.599003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.0.97.61:37167)
0924 11:46:13.599214 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.599223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.599569 (+ 346us) server_negotiation.cc:407] Connection header received
0924 11:46:13.599815 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.599819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.599903 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.600012 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.600717 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47100: BlockingRecv error: recv got EOF from 127.0.0.1:47100 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 11:46:13.734938 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.733309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.0.97.61:37167)
0924 11:46:13.733476 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.733480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.733810 (+ 330us) server_negotiation.cc:407] Connection header received
0924 11:46:13.733960 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.733964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.734049 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.734138 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.734752 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47112: BlockingRecv error: recv got EOF from 127.0.0.1:47112 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:46:13.747908 23390 catalog_manager.cc:1259] Loaded cluster ID: 71c94810644a4e978fb6c0cf612870c7
I20260924 11:46:13.747994 23390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: loading cluster ID for follower catalog manager: success
I20260924 11:46:13.750250 23390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: acquiring CA information for follower catalog manager: success
I20260924 11:46:13.751780 23390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc1e272e3ffc499788d0074a4a4943ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:46:13.999701 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:46:13.997818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.0.97.61:37167)
0924 11:46:13.998042 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:13.998050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:13.998357 (+ 307us) server_negotiation.cc:407] Connection header received
0924 11:46:13.998600 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:13.998604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:13.998689 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:13.998818 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:46:13.999516 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47122: BlockingRecv error: recv got EOF from 127.0.0.1:47122 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:46:14.070457 23438 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: acquiring CA information for follower catalog manager: success
I20260924 11:46:14.072058 23438 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 313e4eee942c424a9b0953d9d56e39d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:46:14.522861 23503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:14.516951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.0.97.61:37167)
0924 11:46:14.517650 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:14.517655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:14.517672 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:46:14.517733 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:14.517737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:14.517879 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:14.517988 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:14.518483 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.519215 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.520817 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.521211 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.521311 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:14.521757 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:14.522256 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:14.522368 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:14.522450 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 11:46:14.522514 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":454,"threads_started":1}
I20260924 11:46:14.534907 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:14.528455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.0.97.60:43623)
0924 11:46:14.529002 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:14.529009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:14.529213 (+ 204us) server_negotiation.cc:407] Connection header received
0924 11:46:14.529462 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:14.529466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:14.529627 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:14.529795 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:14.530195 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.530912 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.532598 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.533026 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.533187 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:14.533693 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:14.534238 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:14.534399 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:14.534513 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:46:14.534602 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":287,"threads_started":1}
I20260924 11:46:14.548192 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:14.540861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.97.60:43623)
0924 11:46:14.541039 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:14.541047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:14.542041 (+ 994us) server_negotiation.cc:407] Connection header received
0924 11:46:14.542274 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:14.542279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:14.542364 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:14.542481 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:14.543183 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.544087 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.546173 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.546780 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.546902 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:14.547489 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:14.547632 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:14.547723 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:14.547823 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 11:46:14.547902 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:46:14.549153 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:14.540702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.97.62:43585)
0924 11:46:14.541467 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:14.541476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:14.542153 (+ 677us) server_negotiation.cc:407] Connection header received
0924 11:46:14.542564 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:14.542571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:14.542788 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:14.542971 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:14.543533 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.544737 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.546642 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.547087 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.547251 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:14.547758 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:14.548224 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:14.548738 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:14.548783 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 11:46:14.548855 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":477,"threads_started":1}
I20260924 11:46:14.549263 23503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:46:14.541884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.0.97.61:37167)
0924 11:46:14.542090 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:46:14.542095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:46:14.543097 (+ 1002us) server_negotiation.cc:407] Connection header received
0924 11:46:14.543346 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:46:14.543351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:46:14.543439 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:46:14.543524 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:46:14.544115 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.545043 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.547135 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:46:14.547738 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:46:14.547842 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:46:14.548475 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:46:14.548595 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:46:14.548666 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 11:46:14.548890 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 11:46:14.548956 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:46:14.552956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 23158
I20260924 11:46:14.554498 23220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:14.660090 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 23158
I20260924 11:46:14.682710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 23232
I20260924 11:46:14.683784 23294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:14.783973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 23232
I20260924 11:46:14.802879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpreJ74/build/asan/bin/kudu pid 23306
I20260924 11:46:14.803895 23368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:46:14.903493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpreJ74/build/asan/bin/kudu with pid 23306
2026-09-24T11:46:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4882 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9245 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (347205 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST